CSS Tokenizer: Comprehensive unit tests

This patch adds comprehensive unit tests to the css-syntax based
tokenizer. I've tried to cover all the edge cases I could think of here.

A few cases don't work correctly, which I've left as commented out
tests:
- Idents starting with two dashes like --abc aren't parsed as idents
- Escape should be replaced with U+FFFD when they represent either
(U+0000), a surrogate codepoint (U+D800 - U+DFFF) or are larger than
the maximum allowed codepoint (U+110000).
- We don't correctly replace U+0000 with U+FFFD
- Our handling of newline escapes (i.e. "check if two code points are a
valid escape") is incorrect in some cases as we don't perform
preprocessing.

Note that there have been minor spec changes since the candidate rec and
the tests here are based on the latest editor's draft.

http://dev.w3.org/csswg/css-syntax

BUG=424988

Review URL: https://codereview.chromium.org/646893003

git-svn-id: svn://svn.chromium.org/blink/trunk@184322 bbb929c8-8fbe-4397-9dbb-9b2b20218538

third_party/WebKit/Source/core/css/parser/CSSParserToken.cpp

@@ -67,41 +67,6 @@ void CSSParserToken::convertToPercentage()
     m_unit = CSSPrimitiveValue::CSS_PERCENTAGE;
 }
 
-// This function is used only for testing
-// FIXME - This doesn't cover all possible Token types, but it's enough for current testing.
-String CSSParserToken::textForUnitTests() const
-{
-    if (!m_value.isNull())
-        return m_value;
-    if (m_type == LeftParenthesisToken)
-        return "(";
-    if (m_type == RightParenthesisToken)
-        return ")";
-    if (m_type == ColonToken)
-        return ":";
-    if (m_type == WhitespaceToken)
-        return " ";
-    if (m_delimiter)
-        return String("'") + m_delimiter + '\'';
-
-    if (m_numericValue) {
-        String unit;
-        if (m_unit == CSSPrimitiveValue::CSS_PERCENTAGE)
-            unit = "%";
-        else if (m_unit == CSSPrimitiveValue::CSS_PX)
-            unit = "px";
-        else if (m_unit == CSSPrimitiveValue::CSS_EMS)
-            unit = "em";
-        else if (m_unit != CSSPrimitiveValue::CSS_NUMBER)
-            unit = "other";
-        if (m_numericValueType == IntegerValueType)
-            return String::number(static_cast<int>(m_numericValue)) + unit;
-        const unsigned fractionalDigits = 6;
-        return String::numberToStringFixedWidth(m_numericValue, fractionalDigits) + unit;
-    }
-    return String();
-}
-
 UChar CSSParserToken::delimiter() const
 {
     ASSERT(m_type == DelimiterToken);

third_party/WebKit/Source/core/css/parser/CSSParserToken.h

@@ -60,7 +60,6 @@ public:
 
     CSSParserTokenType type() const { return m_type; }
     String value() const { return m_value; }
-    String textForUnitTests() const;
 
     UChar delimiter() const;
     NumericValueType numericValueType() const;