Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/wincent/wikitext

🌐 Fast wikitext-to-HTML translator
https://github.com/wincent/wikitext

ruby wiki

Last synced: about 1 month ago
JSON representation

🌐 Fast wikitext-to-HTML translator

Awesome Lists containing this project

README 

        
= Wikitext



The Wikitext extension is a fast wikitext-to-HTML translator written

in C and packaged as a Ruby extension.



Usage is straightforward:



    !!!plain

    $ irb -r wikitext

    >> Wikitext::Parser.new.parse("hello world!")

    => "
hello world!
\n"



= Design goals



I needed a wikitext-to-HTML translator for a Rails application; a number

of design goals flowed on from this:



* _fast_: Rails has a reputation for being slow, so the translator had

  to be part of the solution, not part of the problem

* _efficient_: again, given how much memory Rails likes to use, the

  translator had to be very memory-efficient

* _robust_: on a public-facing web application that had to be up for long

  periods, the translator had to be stable (no crashes, no resource leaks)

* _secure_: again, accepting input from untrusted sources meant that the

  translator had to sanitize or reject unsafe input

* easy to use: for end users, the translator should provide a

  simple, familiar markup as close as possible to what they already know

  from other applications (such as MediaWiki, the wiki software that

  powers Wikipedia)

* _forgiving_: wikitext is presentation markup, not source code, so the

  translator should do a reasonable job of formatting even the most

  invalid markup rather than giving up

* _informative_: when provided invalid markup the translator should

  fail gracefully and emit HTML that provides useful visual feedback

  about where the errors are in the input

* multilingual-friendly: the translator should handle input beyond

  printable ASCII in a compatible fashion

* _attractive_: the emitted HTML source should be consistent and attractive

* valid output: regardless of the input, the translator should

  always produce valid HTML5 output

* well-tested: the translator should have a comprehensive test

  suite to ensure that its behaviour is not only correct but also stable

  over time

* cross-platform: should work identically on Mac OS X, Linux

  (explicitly tested platforms) and perhaps others as well



Some notable things that were _not_ design goals:



* implement _all_ of the MediaWiki syntax (tables etc)



= Markup



The markup is very close to that used by MediaWiki, the most popular wiki

software and the one that powers Wikipedia.



== Headings



    !!!wikitext

    = Heading 1 =

    == Heading 2 ==

    === Heading 3 ===

    ==== Heading 4 ====

    ===== Heading 5 =====

    ====== Heading 6 ======



Are marked up as:



    !!!html

    
Heading 1


    
Heading 2


    
Heading 3


    
Heading 4


    
Heading 5


    
Heading 6



== Paragraphs



Consecutive linebreaks are converted into paragraph breaks.



    !!!wikitext

    This is one paragraph.

    Another line.

    

    And this is another.



Would be marked up as:



    !!!html

    
This is one paragraph. Another line.


    
And this is another.



== Emphasis, Strong



Emphasis is marked up as follows:



    !!!wikitext

    ''emphasized''



Which gets translated into:



    !!!html

    emphasized



Strong is marked up like this:



    !!!wikitext

    '''strong text'''



And transformed into:



    !!!html

    strong text



You can nest spans inside one another, provided you don't try to produce

invalid HTML (for example, nesting strong inside strong). Here is a valid

example:



    !!!wikitext

    '''''foo'' bar''' baz



This would become:



    !!!html

    foo bar baz



Note that the translator emits HTML on the fly, so when it sees the

first run of five apostrophes it has no way of knowing what will come

afterwards and so doesn't know whether you mean to say "strong em" or

"em strong"; it therefore always assumes "strong em". If you wish to

force the alternative interpretation you can do one of the following:



    !!!wikitext

    '' '''foo''' bar'' baz (ie. use whitespace)

    '''''foo''' bar'' baz (ie. insert an empty nowiki span)

    foo bar baz (ie. use explicit HTML tags instead)

    '''foo''' bar baz (ie. use explicit HTML tags instead)



Note that to avoid ambiguity, the translator will not let you intermix

the shorthand style with the literal HTML tag style.



    !!!wikitext

    foo'' (ie. intermixed, invalid)



== Teletype



The translator recognizes both standard HTML +tt+ tags and the

backtick (`) as a shorthand. These two are equivalent:



    !!!wikitext

    fixed

    `fixed`



As of version 2.0, this markup is actually translated to +code+ tags

in the output because the +tt+ tag was removed from the HTML5 standard.



If you need to insert a literal backtick in your text you use a +nowiki+

span:



    !!!wikitext

    here follows a literal ` backtick



To avoid ambiguity, the translator will not let you intermix the two

styles.



== +nowiki+ spans



Already mentioned above, you can use +nowiki+ tags to temporarily disable

wikitext markup. As soon as the translator sees the opening +nowiki+ tag

it starts emitting a literal copy of everything it sees up until the

closing +nowiki+ tag:



    !!!wikitext

    Hello ''world''



Would be emitted as:



    !!!html

    Hello ''world''



== Blockquotes



    !!!wikitext

    > Hello world!

    > Bye for now.



Would be emitted as:



    !!!html

    
Hello world! Bye for now.



You can nest blockquotes or any other kind of block or span inside

blockquotes. For example:



    !!!wikitext

    > first quote

    >> quote inside a quote



== Preformatted text



Any line indented with whitespace will be interpreted as part of a +pre+

block. Wikitext markup inside +pre+ blocks has no special meaning. For

example, consider the following block indented by a single space:



    !!!wikitext

     // source code listing

     void foo(void)

     {

         x++;

     }



Would be translated into:



    !!!html

    
// source code listing

    void foo(void)

    {

        x++;

    }



+pre+ blocks may be nested inside +blockquote+ blocks.



== Internal links



    !!!wikitext

    [[article title]]



Would become:



    !!!html

    article title



And:



    !!!wikitext

    [[title|link text]]



Would become:



    !!!html

    link text



See the Wikitext::Parser attributes documentation for how you can override

the default link prefix (/wiki/ as shown in the example), and how

"red links" can be implemented by applying custom CSS depending on the

link target (this can be used to make links to non-existent pages appear

in a different color).



== Alternative blockquote and preformatted block syntax



For +blockquote+ and +pre+ blocks that go on for many lines it may be

more convenient to use the alternative syntax which uses standard

HTML tags rather than special prefixes at the beginning of each line.



    !!!wikitext

    
This is

    a blockquote!


    

    
And this is

    preformatted text



+blockquote+ and +pre+ blocks may nest inside other +blockquote+

blocks.



Note that to avoid ambiguity, the translator will not let you intermix

the two styles (HTML markup and wikitext markup).



+pre+ blocks may also contain a custom +lang+ attribute for the purposes

of marking up a block for syntax-highlighting (note that the highlighting

itself would be provided by JavaScript in the browser and is not actually

part of the Wikitext extension). For example:



    !!!wikitext

    
puts @person.name



Would be translated into:



    !!!html

    
puts @person.name



The +lang+ attribute may only contain letters, so "Objective-C", for

example would need to be written as "objc" or similar.



== External links



    !!!wikitext

    [http://example.com/ this site]



Would become:



    !!!html

    this site



See the {Wikitext::Parser} attributes documentation for information on

overriding the default external link class (+external+ in this

example), or including a +rel+ attribute of "nofollow" (which may be

useful for search-engine optimization).



Note that in addition to providing a fully-qualified URL including a

protocol (such as "http://" or "ftp://") you also have the option of

using an unqualified "path"-style URL. This is useful for making

links to other pages still on the same site, but outside of the wiki:



    !!!wikitext

    [/issues/1024 ticket #1024]



Would become:



    !!!html

    ticket #1024



Note that no "external" class is included in the generated link.



To avoid false positives, what constitutes a "path" is

narrowly-defined as a string that begins with a slash, optionally

followed by zero or more "path components" consisting of upper and

lowercase letters, numbers, underscores, hyphens or periods. Path

components are separated by a slash, and the trailing slash after

the last path component is optional.



== Images



    !!!wikitext

    {{foo.png}}



When outputting using HTML syntax (the default), this would become:



    !!!html

    foo.png



When outputting using XML syntax, this would become a self-closing tag:



    !!!html

    foo.png



See the Wikitext::Parser documentation for information on setting the

output syntax.



You can override the "/images/" prefix using the +img_prefix+ attribute

of the Parser.



You can also specify "absolute" image "src" attributes regardless of

the current prefix setting by starting the image path with a forward

slash; that is:



    !!!wikitext

    {{/foo.png}}



Would become:



    !!!html

    /foo.png



== Lists



Lists come in both unordered ("ul"):



    !!!wikitext

    * item

    * item

    * item



And ordered ("ol") forms:



    !!!wikitext

    # first

    # second

    # third



These would produce, respectively:



    !!!html

    
    

          
  • item
    • 

          
  • item
    • 

          
  • item
    • 

        



And:



    !!!html

    
    

          
  1. first
    2. 

          
  2. second
    3. 

          
  3. third
    4. 

        



Lists may be nested inside one another as needed. For example:



    !!!wikitext

    # outer a

    # outer b

    #* nested 1

    #* nested 2

    # outer c

    ## nested foo

    ## nested bar

    ##* x

    ##* y

    ##** z



Would produce:



    !!!html

    
    

          
  1. outer a
    2. 

          
  2. outer b

            
      

                
    • nested 1
      • 

                
    • nested 2
      • 

              
    

          
    3. 

          
  3. outer c

            
      

                
    1. nested foo
      2. 

                
    2. nested bar

                  
        

                      
      • x
        • 

                      
      • y

                        
          

                            
        • z
          • 

                          
        

                      
        • 

                    
      

                
      3. 

              
    

          
    4. 

        



= Ruby support



Version 4.0.0 and above target Ruby 2.0.0 or higher.



For older versions of Ruby, you may use the 3.1 release or older.



= Rails support



The Wikitext extension provides a template handler so that templates named

following the template_name.html.wikitext format will automatically be

translated from wikitext markup into HTML when rendered.



Additionally, an optional Haml filter is available if you require

"wikitext/haml_filter", which enables you to write wikitext markup inline

(in Haml):



    :wikitext

      = Here is some [[wikitext]] =



Likewise, a +to_wikitext+ method (aliased as +w+) is added to the +String+

class (and also +NilClass+, for convenience) so that content can be easily

translated from inside view templates following patterns like:



    @post.body.w



The +to_wikitext+ method will preprocess its string using the

String#wikitext_preprocess method, if it is defined, before feeding the string

through the parser. This can be used to add application-specific behavior such

as converting special strings like:



    !!!wikitext

    ticket #1234



into links. An example preprocessor is included with the extension but it is

not active by default; it can be activated with:



    require 'wikitext/preprocess'



Finally, a Wikitext::Parser#shared_parser method is added to provide

convenient access to a shared singleton instance of the parser so as to

avoid repeatedly instantiating and setting up new parser instances as part

of every request.



== Rails 2.3



For Rails 2.3.x support, use version 2.1.x of the Wikitext gem.



The plug-in can be activated with an appropriate config.gem statement

in your config/environment.rb:



    config.gem 'wikitext', '2.1.1'



== Rails 3.0



For Rails 3.0.x support, use version 2.1.x of the Wikitext gem.



Add a line like the following to your Gemfile:



    gem 'wikitext', '~> 2.1.1'



Note that while older versions of Wikitext do work with Rails 3 to some degree,

for full compatibility Wikitext version 2.0 or higher should be used.



== Rails 3.1



Add a line like the following to your Gemfile:



  gem 'wikitext'



= Links



* Website: http://wincent.github.io/wikitext/

* RDoc: http://wincent.github.io/wikitext/

* Source: https://github.com/wincent/wikitext



= Author



Wikitext is written and maintained by Greg Hurrell ([email protected]).

Other contributors that have submitted patches include:



* Mike Stangel



= License



Copyright 2007-present Greg Hurrell. All rights reserved.



Redistribution and use in source and binary forms, with or without

modification, are permitted provided that the following conditions are met:



1. Redistributions of source code must retain the above copyright notice,

   this list of conditions and the following disclaimer.

2. Redistributions in binary form must reproduce the above copyright notice,

   this list of conditions and the following disclaimer in the documentation

   and/or other materials provided with the distribution.



THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"

AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE

IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE

ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR CONTRIBUTORS BE

LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR

CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF

SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS

INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN

CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)

ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE

POSSIBILITY OF SUCH DAMAGE.



= Feedback



Please let me know if you're using the Wikitext extension in your project.

If you have any bug reports or feature requests please open a ticket in

the issue tracker at https://github.com/wincent/wikitext/issues.