Overview
The problem can happen in v5.0 or v5.1 when you have a test case (or test set) folder that has either a recursive link to one of its own children as parents, or where you hit a bug in the system that would it create hundreds of incorrect child folders. Both of these issues were fixed in later versions of v5, but there is a solution if you are using an older version and run into the issue.
Solution
The solution is to make all the folders "top-level" root folders. That will get rid of the error message, so that you can then use the application UI to delete any folders that you don't actually want.
The SQL command to run for test cases is:
UPDATE TST_TEST_CASE_FOLDER SET PARENT_TEST_CASE_FOLDER_ID = NULL WHERE PROJECT_ID = <project id>
For test sets it is:
UPDATE TST_TEST_SET_FOLDER SET PARENT_TEST_SET_FOLDER_ID = NULL WHERE PROJECT_ID = <project id>
In both cases, once this is done, you will need to manually fix the hierarchy to have the proper parent...child folders.
If you are a cloud hosted customer, you would have raised a ticket and our support team would do this for you.