"treat line breaks as whitespace - precedent, good for Hebrew" -- How is that? I run a Hebrew wiki and we treat line break as <br>. --Yonat
Why linebreaks are evil? #
I've been meaning to pick this subject for a long time, but I don't want to present my arguments in wrong way, or to miss any important argument. So I was preparing for this carefully. This is going to be a rant against converting newlines to <br>, as you can easily guess. Ok, lets begin.
Target audience #
The justification of the current handling of newlines I've heard goes something like this: "this is intuitive, people coming to wikis from MS Word or blogging software will expect it".
So new wiki users will be happy to have this feature in Creole, as it means less new things for them, right?
In the meantime, people who already use wikis, will stumble on it and find it awkward at least as much as me. And it's not just some time until they get used to it -- because it's not a "standard" they can get used to. 99% of wikis that don't use Creole handle newlines as spaces. Many web forums, message boards, less advanced blogging software handle newlines like that too -- simply because that's how HTML does it, and because it's easy to translate such text directly into HTML. Practically all proffessional or half-professional typesetting languages treat single newlines the same as space -- from HTML, through PostScript and Rich Text Format, up to LaTeX and TeX. Single newline has no meaning other than simple space. And, byt the way, the same goes for handling multiple spaces as a single one (although RTF is different in this regard).
This means, that while new users are happily typing their text the way they think it should work, the experienced users need to maintain this "split personality", remembering where they can use enter, and where they can't, because it will produce invalid layout (mind you, there are literally 2 or 3 cases when a newline is actually considered correct and needed in typesetting).
Now, the oldies will eventually die off, we need to look with hope at the new generation of blogg^H^H^H^H^Hwikizens. But even they, as they will discover more advanced software, will have to maintain the "shizm" between how the newlines are handled. And it's in an especially sensitive, "transition" time, when a small obstacle like that can make them give up and forever stay in the newbie world of MS Word and blogs, locked out from proffessional software and non-Creole wikis.
Usability experts know of a thing called "myth of experienced user" -- a trap that interface designers often fall for, by designing two intrefaces, one for newbies and one for experienced users -- thinking that when a newbie will become experienced, they can switch to the more powerful but also more complicated interface. But this never happens, and users are locked forever in the "newbie" interface, just because they don't become more experienced with the more powerful one by using the less complicated one.
I'm strongly convinced that this kind of "pro-newbie" decissions create a similar chasm -- not between interfaces of a single application, but between different wikis. Creole is not intended to be the one and only wiki markup. We don't want to lock users from other wikis, we want to introduce them gently to them. Thus, Creole should be simple and easy to learn, but it should not be substantially different from other wiki markups.
Technical difficulties #
Everyone who tried to implement a Creole parser knows that this rule, together with some other special cases, increases the parser's complexity considerably, making it much harder to create, debug and extend. Hard to write parser means worse adoption across different wiki engines and more accidental incosistences between implementations. That's on the side of the developers.
Difficulties on the side of users usually involve copying and pasting of text from their e-mails or text editor -- different line-lengths will result in text that looks like this:
Lorem ipsum dolor sit amet, consectetuer adipiscing elit, sed diam nonummy nibh euismod tincidunt ut laoreet dolore magna aliquam erat volutpat. Ut wisi enim ad minim veniam, quis nostrud exerci tation ullamcorper suscipit lobortis nisl ut aliquip ex ea commodo consequat.
That's the effect of mixing of the browser's automatic line wrapping with the user's manual one. The same thing happens when wiki's textarea has different width than the rendered page -- when a wiki site has a large sidebar, for example.
The wiki admins will also have a hard time with this. Text produced by the wiki users will practically lock them in one layout, because changing (especially decreasing) the line length will lead to the effect above.
Even worse, for "flowing" layouts, users that have non-standard font size or window size will experience the effect above.
I don't even want to think about printing wiki pages.
Now, cleaning up this mess automatically is impossible (while it is possible to convert the text the other way around) -- it involves manually browsing trough the text and removing all the spuriouous newlines. I've done it several times. It's debilitating.
Often just looking* for the spurious newlines is hard -- because with the automatically wrapped textarea they are InvisibleMarkup. Another reason to drop them.
Solution #
Remove the rule about newlines from the Creole specification. Add a separate rule for forcing a linebreak when it is really absolutely required -- something like "
" or "##" or "||" at the end of the line, for example.
If we absolutely need to accomodate the blog users, then just don't specify the newline handling in the spec -- allow different wikis to use what is best for their user base. But I'm all for specifying that single newlines shall be ignored and treated as spaces.
-- RadomirDopieralski, 2006-12-11
I found some surveys about how different wiki engines treat newlines on MeatBall: http://www.usemod.com/cgi-bin/mb.pl?ParagraphFormattingRules
Shall we start a discussion on ChangeLinebreakMarkupProposal?
To start moving this fowrward, I'd like to propose "\\\s*$" as the regular expression for forcing line breaks. I'ts simple to type, looks similar to the markup used for marking newlines in many markup langueges, and is consistent with use of "\" for as an escape character.
The requirement for an end of line makes this markup more WYSIWYG, but makes it inadequate to use in table cells. "\\(\s*$|\s)" could be used instead...
-- RadomirDopieralski, 2007-01-01
For me, the difference between a break line and a new paragraph is the space between the two different lines. Am I wrong? Is there a semantic difference?
So, my first thought was to go even further: only one newline is sufficient to change *paragraphs* (<p></p>). Two or more newlines would be treated as one and only one paragraph change. Food for thoughts...
A newline is considered as a new paragraph (marked by a pilcrow) in Microsoft Word. In Word 2007, the spacing between paragraphs is now obvious in the default template (10 points after a paragraph).
Break line #
To break a line (<br /> ), any \s+\\\\\s+} (that's two backslashes (\\) preceded and followed by at least one whitespace) would be sufficient in my opinion. The rendering engine would eliminate theses spaces.
Then, the questions are:
- Why do we *absolutely* need two newlines for paragraphs? (The spec could say: At least one newline is necessary to create a new paragraph.
- Is one backslash sufficient to break a line?
- Do we need to put a newline after or one whitespace is sufficient?
- What about one whitespace before?
- Should two or more subsequent whitespaces be treated as only one space?
Remarks:
- The backslash key is not easy to find on several keyboard layouts.
-- EricChartre, 2007-01-03
Good questions, and I'm glad you asked them, so that we can explicitly substantiate the choices made so far.
I will start with double newline for separating paragraphs. This comes from a long tradition of message boards, newsgroups, various faqs and rfcs, walktroughs and similar text documents -- they are usually preformatted, with single line breaks used merely to fold the lines of text, and an empty line to separate paragraphs. This "tradition" is also very practical in more formal cases, where you can use newlines to emphasize the structure of the raw source code, while keeping the rendering independent -- the "double newline" rule is present in TeX and many markup languages derieved from it, including markup of practically all the wiki engines.
The textareas used commonly to edit wiki markup are usually very simple and pretty hard to use. Until recently, they didn't even have any support for wrapping the text -- if you had any long lines, you had to scroll. Today's browsers are a little better in this regard, but only minimally -- line wrapping is pretty much broken in most of them -- thus providing some means to manually control the flow of text in the source, without impact to the rendering, is still important.
Even if one uses an external editor to edit text -- and there are both special browser plugins or editor scripts allowing to do that -- one's not free from the line-wrapping problems. Many text editors will wrap text by default (which is useful for writing e-mails, for example). You also might need to put on the wiki pages some text taken from e-mails, newsposts, various text files, web pages (some browsers will reformat text when it's copied), or even scanned text. Ignoring sigle newlines allows you to minimize the work with reformatting such text (adding newlines for several paragraphs is always easier than removing them for several hundred lines).
There is also a question of presentation -- with "modern" line-wrapping textareas, a single newline is effectively InvisibleMarkup if it comes near the edge of the editor area. Tracking down and removing such "spurious" newlines is not easy and usually pretty annoying.
As for treating consecutive whitespace as single space, it's similar deal. Plus, we don't want people to use spaces to indent or center text -- it's not only ugly and hard to maintain -- it's also totally unportable to devices/software/sites using different fonts and screen widths. Of course, any space at the end of lines should be ignoed because it forms InvisibleMarkup.
Now, as to my propostion of the forced line break markup, it's not really supposed to be a backslash and a space -- rather it's a backslash at the end of the line. The whitespace is added for the reasons mentioned above. Note, that this "fits" with the use of backslash for an escape character -- "eascaped" newline is being treated literally. Using double backslash has the problem of conflicting with the markup of eascaped literal backslash.
I'm think that requiring a newline in the markup for forced line break where possible is a good idea -- it makes the code more WYSIWYG and thus ReadableMarkup -- you don't need to scan whole line to spot linebreaks -- you only need to look at the actual ends of lines in the raw code. Of course this has a problem if there are places where the newline is forbidden -- I think it's an indication of a greater problem in the markup, though.
-- RadomirDopieralski, 2006-01-03
Good points.
So, to summarize your proposition with my own:
Rule #1: Paragraphs are separated by at least two newlines (CR/LF, LF or FF). Any number of whitespaces before, in between or after will be ignored. More than two consecutive newlines will be considered as one and only one change of paragraph.
New paragraph: /(\s*\n\s*\n\s*)/s (Perl expression with the /s modifier)
Rule #2: To break a line, one must escape a newline. The rule should be permissive enough to allow any number of whitespaces (including CR/LF or LF) after the backslash. A space before is not necessary. Rule of least surprise.
Line break: /(\s*\\\s*\n\s*)/s (Perl expression with the /s modifier)
Rule #3: Tabs are treated as spaces.
Rule #4: Consecutive spaces or tabs are treated as one space only.
Rule #5: Spaces or tabs at the beginning of a line are ignored.
What about FF (\f) or vertical tabs (\v)?
-- EricChartre, 2007-01-05
I would say the "new paragraph" in rule #1 is rather /(\s*\n(\s*\n)+)/s. The rule #2 would be much simplier to understand if we just stripped all spaces and tabs from the end of lines before parsing. I don't know where you got #5 from, it wasn't mentioned anywhere -- treating indented lines differently than non-indented ones is still an option -- just not used anywahere. One point, though, is that the amount of indentation shouldn't be relevant (no space counting). The form feed and vertical tab characters are treated as space -- you have the vertical line and headings to give structure to your text.
-- RadomirDopieralski, 2007-01-05
It would be easier, indeed, to strip all consecutive whitespaces and spaces from the end of lines before parsing. Most regular expressions would be simpler and we wouldn't have to use the /s modifier. However, wouldn't it create the need for a new parsing pass?
I took some creative liberty for #5 ;-) while I was on the subject of spaces. See the talk about quoting.
-- EricChartre, 2007-01-05
Looking at the "characters not in Creole" at Terms, I can imagine using several others characters for forcing line end:
- # or some combination of it (possible conflict with wikis using ##...## markup).
- $ or some combination of it (possible conflict with wikis using $$...$$ markup).
- = or some combination of it, possibly only at the end of line
- | or some combination, would make it impossible to use inside table cells
Note, that we probably don't need or want a markup for forcing a break of an empty line.
So, any ideas?
-- RadomirDopieralski, 2007-01-05
Consider %, too. If %%% is used for breaking lines, as in PhpWiki, it would be NotNew.
-- MicheleTomaiuolo, 2007-01-05