The relational model has a formal definition given by E.F. Codd that is founded in mathematical theory, more information about which can as always be found on Wikipedia.

"a relation is a data structure which consists of a heading and an unordered set of tuples which share the same type."

So consider something that does not meet that definition: data which you place under one heading, but do not share the same type. This certainly sounds like a document database, or a key-value store - where "value" is implicitly of an undefined type.

Interestingly but perhaps less relevant, an ordered set should not be considered relational, which is why for example you should not consider that adding a clustered index on a table will guarantee the order in which rows are returned by a query even though this may occur as a side effect of the underlying implementation.

Note also, that you can see most RDBMSs are - perhaps fortunately - fairly lax about limiting themselves to theory; see ORDER BY in your SQL implementation of choice.