flexmark java
1.0.0
This document provides a comprehensive overview of flexmark-java, a high-performance Java implementation of the CommonMark Markdown parser. It details its features, including speed, extensibility, and detailed source position tracking, along with instructions for quick start, available extensions, and migration from pegdown. The document also covers its ongoing development and contribution guidelines.
flexmark-java
flexmark-java is a Java implementation of CommonMark (spec 0.28) parser using the
blocks first, inlines after Markdown parsing architecture.
Its strengths are speed, flexibility, Markdown source element based AST with details of the
source position down to individual characters of lexemes that make up the element and
extensibility.
The API allows granular control of the parsing process and is optimized for parsing with a large
number of installed extensions. The parser and extensions come with plenty of options for parser
behavior and HTML rendering variations. The end goal is to have the parser and renderer be able
to mimic other parsers with great degree of accuracy. This is now partially complete with the
implementation of Markdown Processor Emulation
Motivation for this project was the need to replace pegdown parser in my Markdown Navigator
plugin for JetBrains IDEs. pegdown has a great feature set but its speed in general is less
than ideal and for pathological input either hangs or practically hangs during parsing.
️Version 0.60.0 has breaking changes due to re-organization, renaming, clean up and
optimization of implementation classes. Changes are detailed in
Version-0.60.0-Changes.
latest
Requirements
For Versions 0.62.2 or below, Java 8 or above, Java 9+ compatible. For Versions 0.64.0 or
above, Java 11 or above.
The project is on Maven: com.vladsch.flexmark
The core has no dependencies other than org.jetbrains:annotations:24.0.1. For extensions, see
extension description below.
The API is still evolving to accommodate new extensions and functionality.
Quick Start
For Maven, add flexmark-all as a dependency which includes core and all modules to the
following sample:
Source:
BasicSample.java
Building via Gradle
Building with Android Studio
Additional settings due to duplicate files:
More information can be found in the documentation:
Wiki Home Usage Examples
Extension Details
Writing Extensions
Pegdown Migration Helper
PegdownOptionsAdapter class converts pegdown Extensions.* flags to flexmark options and
extensions list. Pegdown Extensions.java is included for convenience and new options not found
in pegdown 1.6.0. These are located in flexmark-profile-pegdown module but you can grab the
source from this repo: PegdownOptionsAdapter.java, Extensions.java and make your own
version, modified to your project's needs.
You can pass your extension flags to static PegdownOptionsAdapter.flexmarkOptions(int) or you
can instantiate PegdownOptionsAdapter and use convenience methods to set, add and remove
extension flags. PegdownOptionsAdapter.getFlexmarkOptions() will return a fresh copy of
DataHolder every time with the options reflecting pegdown extension flags.
Default flexmark-java pegdown emulation uses less strict HTML block parsing which interrupts an
HTML block on a blank line. Pegdown only interrupts an HTML block on a blank line if all tags in
the HTML block are closed.
To get closer to original pegdown HTML block parsing behavior use the method which takes a
boolean strictHtml argument:
A sample with a custom link resolver is also available, which includes link resolver for
changing URLs or attributes of links and a custom node renderer if you need to override the
generated link HTML.
flexmark-java has many more extensions and configuration options than
pegdown in addition to extensions available in pegdown 1.6.0.
Available Extensions via PegdownOptionsAdapter
Latest Additions and Changes
Major reorganization and code cleanup of implementation in version 0.60.0, see
Version-0.60.0-Changes thanks to great work by
Alex Karezin you can get an overview of module dependencies
with ability to drill down to packages and classes.
Merge API to merge multiple markdown documents into a single
document.
Docx Renderer Extension:
Limited Attributes Node Handling
Extensible HTML to Markdown Converter module:
flexmark-html2md-converter. Sample:
HtmlToMarkdownCustomizedSample.java
Java9+ module compatibility
Compound Enumerated References
Enumerated References Extension for creating
legal numbering for elements and headings.
Macros Extension to allow arbitrary markdown content to be
inserted as block or inline elements, allowing block elements to be used where only inline
elements are allowed by syntax.
GitLab Flavoured Markdown for parsing and
rendering GitLab markdown extensions.
OSGi module courtesy Dan Klco (GitHub @klcodanr)
Media Tags Media link transformer extension courtesy
Cornelia Schultz (GitHub @CorneliaXaos) transforms links
using custom prefixes to Audio, Embed, Picture, and Video HTML5 tags.
Translation Helper API to make translating markdown
documents easier.
Admonition To create block-styled side content. For
complete documentation please see the Admonition Extension, Material for MkDocs
documentation.
Enumerated Reference to create enumerated
references for figures, tables and other markdown elements.
Attributes to parse attributes of the form {name name=value name='value' name="value" #id .class-name} attributes.
YouTube Embedded Link Transformer
thanks to Vyacheslav N. Boyko (GitHub @bvn13) transforms simple links to youtube videos to
embedded video iframe HTML.
Docx Converter using the docx4j library. How to use:
DocxConverter Sample, how to customize:
Customizing Docx Rendering
Development of this module was sponsored by
Johner Institut GmbH.
Update library to be CommonMark (spec 0.28) compliant and add
ParserEmulationProfile.COMMONMARK027 and ParserEmulationProfile.COMMONMARK028 to allow
selecting a specific spec version options.
Custom node rendering API with ability to invoke standard rendering for an overridden node,
allowing custom node renders that only handle special cases and let the rest be rendered as
usual. custom link resolver
Gfm-Issues and
Gfm-Users extensions for parsing and rendering #123 and
@user-name respectively.
Deep HTML block parsing option for better handling of raw text tags that come after other tags
and for pegdown HTML block parsing compatibility.
flexmark-all module that includes: core, all extensions, formatter, JIRA and YouTrack
converters, pegdown profile module and HTML to Markdown conversion.
PDF Output Module
PDF output using Open HTML To PDF
Typographic implemented
XWiki Macro Extension
Jekyll Tags
Html To Markdown
Maven Markdown Page Generator Plugin
Markdown Formatter module to output AST as markdown with
formatting options.
Tables for Markdown Formatter
with column width and alignment of markdown tables:
Releases, Bug Fixes, Enhancements and Support
I use flexmark-java as the parser for Markdown Navigator plugin for JetBrains IDEs. I tend to
use the latest, unreleased version to fix bugs or get improvements. So if you find a bug that is
a show stopper for your project or see a bug in github issues page marked fixed for next release that is affecting your project then please let me know and I may be able to promptly
make a new release to address your issue. Otherwise, I will let bug fixes and enhancements
accumulate thinking no one is affected by what is already fixed.
Extension points in the API are many and numerous
There are many extension options in the API with their intended use. A good soft-start is the
flexmark-java-samples module which has simple samples for asked for
extensions. The next best place is the source of an existing extension that has similar syntax
to what you want to add.
If your extension lines up with the right API, the task is usually very short and sweet. If your
extension uses the API in an unintended fashion or does not follow expected housekeeping
protocols, you may find it an uphill battle with a rat's nest of if/else condition handling and
fixing one bug only leading to creating another one.
Generally, if it takes more than a few dozen lines to add a simple extension, then either you
are going about it wrong or the API is missing an extension point. If you look at all the
implemented extensions you will see that most are a few lines of code other than boiler plate
dictated by the API. That is the goal for this library: provide an extensible core that makes
writing extensions a breeze.
The larger extensions are flexmark-ext-tables and flexmark-ext-spec-example, the meat of
both is around 200 lines of code. You can use them as a guide post for size estimating your
extension.
My own experience adding extensions shows that sometimes a new type of extension is best
addressed with an API enhancement to make its implementation seamless, or by fixing a bug that
was not visible before the extension stressed the API in just the right way. Your intended
extension may just be the one requiring such an approach.
Don't hesitate to open an issue if you can't find the answer
The takeaway is: if you want to implement an extension or a feature please don't hesitate to
open an issue and I will give you pointers on the best way to go about it. It may save you a lot
of time by letting me improve the API to address your extension's needs before you put a lot of
fruitless effort into it.
I do ask that you realize that I am chief cook and bottle washer on this project, without an
iota of Vulcan Mind Melding skills. I do ask that you describe what you want to implement
because I can't read your mind. Please do some reconnaissance background work around the source
code and documentation because I cannot transfer what I know to you, without your willing
effort.
Consulting is available
If you have a commercial application and don't want to write the extension(s) yourself or want
to reduce the time and effort of implementing extensions and integrating flexmark-java, feel
free to contact me. I am available on a consulting/contracting basis.
Markdown Processor Emulation
Despite its name, commonmark is neither a superset nor a subset of other markdown flavors.
Rather, it proposes a standard, unambiguous syntax specification for the original, "core"
Markdown, thus effectively introducing yet another flavor. While flexmark is by default
commonmark compliant, its parser can be tweaked in various ways. The sets of tweaks required to
emulate the most commonly used markdown parsers around are available in flexmark as
ParserEmulationProfiles.
As the name ParserEmulationProfile implies, it's only the parser that is adjusted to the
specific markdown flavor. Applying the profile does not add features beyond those available in
commonmark. If you want to use flexmark to fully emulate another markdown processor's behavior,
you have to adjust the parser and configure the flexmark extensions that provide the additional
features available in the parser that you want to emulate.
A rewrite of the list parser to better control emulation of other markdown processors as per
Markdown Processors Emulation is complete. Addition of
processor presets to emulate specific markdown processing behaviour of these parsers is on a
short to do list.
Some emulation families do a better better job of emulating their target than others. Most of
the effort was directed at emulating how these processors parse standard Markdown and list
related parsing specifically. For processors that extend original Markdown, you will need to add
those extensions that are already implemented in flexmark-java to the Parser/Renderer builder
options.
Extensions will be modified to include their own presets for specific processor emulation, if
that processor has an equivalent extension implemented.
If you find a discrepancy please open an issue so it can be addressed.
Major processor families are implemented and some family members also:
profiles to encapsulate configuration details for variants within the
family were added in 0.11.0:
History and Motivation
flexmark-java is a fork of commonmark-java project, modified to generate an AST which
reflects all the elements in the original source, full source position tracking for all elements
in the AST and easier JetBrains Open API PsiTree generation.
The API was changed to allow more granular control of the parsing process and optimized for
parsing with a large number of installed extensions. The parser and extensions come with many
tweaking options for parser behavior and HTML rendering variations. The end goal is to have the
parser and renderer be able to mimic other parsers with great degree of accuracy.
Motivation for this was the need to replace pegdown parser in Markdown Navigator plugin.
pegdown has a great feature set but its speed in general is less than ideal and for
pathological input either hangs or practically hangs during parsing.
commonmark-java has an excellent parsing architecture that is easy to understand and extend.
The goal was to ensure that adding source position tracking in the AST would not change the ease
of parsing and generating the AST more than absolutely necessary.
Reasons for choosing commonmark-java as the parser are: speed, ease of understanding, ease of
extending and speed. Now that I have reworked the core and added a few extensions I am extremely
satisfied with my choice.
Another goal was to improve the ability of extensions to modify parser behavior so that any
dialect of markdown could be implemented through the extension mechanism. An extensible options
API was added to allow setting of all options in one place. Parser, renderer and extensions use
these options for configuration, including disabling some core block parsers.
This is a work in progress with many API changes. No attempt is made to keep backward API
compatibility to the original project and until the feature set is mostly complete, not even to
earlier versions of this project.
Feature Comparison
(1)
flexmark-java pathological input of 100,000 [ parses in 68ms, 100,000 ] in 57ms, 100,000
nested [ ] parse in 55ms
(2)
commonmark-java pathological input of 100,000 [ parses in 30ms, 100,000 ] in 30ms, 100,000
nested [ ] parse in 43ms
(3)
pegdown pathological input of 17 [ parses in 650ms, 18 [ in 1300ms
Progress
I am very pleased with the decision to switch to commonmark-java based parser for my own
projects. Even though I had to do major surgery on its innards to get full source position
tracking and AST that matches source elements, it is a pleasure to work with and is now a
pleasure to extend. If you don't need source level element AST or the rest of what flexmark-java
added and CommonMark is your target markdown parser then I encourage you to use
commonmark-java as it is an excellent choice for your needs and its performance does not
suffer for the overhead of features that you will not use.
Benchmarks
Latest, Jan 28, 2017 flexmark-java 0.13.1, intellij-markdown from CE EAP 2017, commonmark-java
0.8.0:
Ratios of above:
Because these two files represent the pathological input for pegdown, I no longer run them as
part of the benchmark to prevent skewing of the results. The results are here for posterity.
Ratios of above:
Contributing
Pull requests, issues and comments welcome ?. For pull requests:
License
Copyright (c) 2015-2016 Atlassian and others.
Copyright (c) 2016-2023, Vladimir Schneider,
BSD (2-clause) licensed, see LICENSE.txt file.
example:
Mitsuba Renderer 3
|
Documentation
|
Tutorial videos
|
Linux
|
MacOS
|
Windows
|
PyPI
|
|---|---|---|---|---|---|
️
Warning
️
There currently is a large amount of undocumented and unstable work going on in
the master branch. We'd highly recommend you use our
latest release
until further notice.
If you already want to try out the upcoming changes, please have a look at
this porting guide.
It should cover most of the new features and breaking changes that are coming.
Introduction
Mitsuba 3 is a research-oriented rendering system for forward and inverse light
transport simulation developed at EPFL in Switzerland.
It consists of a core library and a set of plugins that implement functionality
ranging from materials and light sources to complete rendering algorithms.
Mitsuba 3 is retargetable: this means that the underlying implementations and
data structures can transform to accomplish various different tasks. For
example, the same code can simulate both scalar (classic one-ray-at-a-time) RGB transport
or differential spectral transport on the GPU. This all builds on
Dr.Jit, a specialized just-in-time(JIT) compiler developed specifically for this project.
Main Features
Cross-platform: Mitsuba 3 has been tested on Linux (x86_64), macOS
(aarch64, x8664), and Windows (x8664).
High performance: The underlying Dr.Jit compiler fuses rendering code
into kernels that achieve state-of-the-art performance using
an LLVM backend targeting the CPU and a CUDA/OptiX backend
targeting NVIDIA GPUs with ray tracing hardware acceleration.
Python first: Mitsuba 3 is deeply integrated with Python. Materials,
textures, and even full rendering algorithms can be developed in Python,
which the system JIT-compiles (and optionally differentiates) on the fly.
This enables the experimentation needed for research in computer graphics and
other disciplines.
Differentiation: Mitsuba 3 is a differentiable renderer, meaning that it
can compute derivatives of the entire simulation with respect to input
parameters such as camera pose, geometry, BSDFs, textures, and volumes. It
implements recent differentiable rendering algorithms developed at EPFL.
Spectral & Polarization: Mitsuba 3 can be used as a monochromatic
renderer, RGB-based renderer, or spectral renderer. Each variant can
optionally account for the effects of polarization if desired.
Tutorial videos, documentation
We've recorded several YouTube videos that provide a gentle introduction
Mitsuba 3 and Dr.Jit. Beyond this you can find complete Juypter notebooks
covering a variety of applications, how-to guides, and reference documentation
on readthedocs.
Installation
We provide pre-compiled binary wheels via PyPI. Installing Mitsuba this way is as simple as running
pip install mitsuba
on the command line. The Python package includes thirteen variants by default:
scalar_rgb
scalar_spectral
scalarspectralpolarized
llvmadrgb
llvmadmono
llvmadmono_polarized
llvmadspectral
llvmadspectral_polarized
cudaadrgb
cudaadmono
cudaadmono_polarized
cudaadspectral
cudaadspectral_polarized
The first two perform classic one-ray-at-a-time simulation using either a RGB
or spectral color representation, while the latter two can be used for inverse
rendering on the CPU or GPU. To access additional variants, you will need to
compile a custom version of Dr.Jit using CMake. Please see the
documentation
for details on this.
Requirements
Python >= 3.8
(optional) For computation on the GPU: Nvidia driver >= 495.89
(optional) For vectorized / parallel computation on the CPU: LLVM >= 11.1
Usage
Here is a simple "Hello World" example that shows how simple it is to render a
scene using Mitsuba 3 from Python:
# Import the library using the alias "mi"import mitsuba as mi# Set the variant of the renderermi.setvariant('scalarrgb')# Load a scenescene = mi.loaddict(mi.cornellbox())# Render the sceneimg = mi.render(scene)# Write the rendered image to an EXR filemi.Bitmap(img).write('cbox.exr')
Tutorials and example notebooks covering a variety of applications can be found
in the documentation.
About
This project was created by Wenzel Jakob.
Significant features and/or improvements to the code were contributed by
Sébastien Speierer,
Nicolas Roussel,
Merlin Nimier-David,
Delio Vicini,
Tizian Zeltner,
Baptiste Nicolet,
Miguel Crespo,
Vincent Leroy, and
Ziyi Zhang.
When using Mitsuba 3 in academic projects, please cite:
@software{Mitsuba3,title = {Mitsuba 3 renderer},author = {Wenzel Jakob and Sébastien Speierer and Nicolas Roussel and Merlin Nimier-David and Delio Vicini and Tizian Zeltner and Baptiste Nicolet and Miguel Crespo and Vincent Leroy and Ziyi Zhang},note = {https://mitsuba-renderer.org},version = {3.1.1},year = 2022} 
