Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: Files displays broken encoding when ⚰ is added to Markdown file #766

Open
5 of 8 tasks
code-surfer opened this issue Aug 17, 2023 · 3 comments
Open
5 of 8 tasks

Comments

@code-surfer
Copy link

⚠️ This issue respects the following points: ⚠️

Bug description

After upgrading nextcloud to 26.0.5, I noticed that some Markdown files are shown with broken encoding in the editor. They were fine in many nextcloud versions up to 23.0.12. I updated in steps to 26.0.5. I also updated to 27.0.2, but the bug persists.

All files are encoded in UTF-8.
I've tracked it down to: when I add a character ⚰ (U+26B0) it breaks.
With äöü in the file, it displays correctly.
With äöü⚰ in the file, it displays as äöü⚰
The files on the filesystem are fine. It's a problem with the nextcloud files editor.

I've used docker images nextcloud:23-fpm .. nextcloud:27-fpm

Steps to reproduce

  1. In Files, create a test.md file
  2. Click it to open it in the editor
  3. Add test äöü
  4. Close the file
  5. Open it again: it displays correctly
  6. Add . Still looking fine
  7. Close the file
  8. Open it again: it displays test äöü ⚰

Expected behavior

The file show display correctly as it did in older nextcloud versions.

Installation method

Community Docker image

Nextcloud Server version

27

Operating system

Debian/Ubuntu

PHP engine version

Other

Web server

Nginx

Database engine version

MariaDB

Is this bug present after an update or on a fresh install?

Upgraded to a MAJOR version (ex. 22 to 23)

Are you using the Nextcloud Server Encryption module?

None

What user-backends are you using?

  • Default user-backend (database)
  • LDAP/ Active Directory
  • SSO - SAML
  • Other

Configuration report

{                 
    "system": {                                     
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [     
            "nextcloud.timesink.de",
            "nc.timesink.de"                                 
        ],                                             
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "overwrite.cli.url": "https:\/\/nextcloud.timesink.de",                                                             
        "dbtype": "mysql",                               
        "version": "27.0.2.1", 
        "dbname": "***REMOVED SENSITIVE VALUE***",       
        "dbhost": "***REMOVED SENSITIVE VALUE***",           
        "dbport": "",                       
        "dbtableprefix": "oc_",     
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "logtimezone": "UTC", 
        "instanceid": "***REMOVED SENSITIVE VALUE***",                                                                                                                                                                                            [0/28]
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",   
        "trusted_domains": [                                                                                                
            "nextcloud.timesink.de",
            "nc.timesink.de"           
        ],                             
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "overwrite.cli.url": "https:\/\/nextcloud.timesink.de",                                                             
        "dbtype": "mysql",     
        "version": "27.0.2.1", 
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",                       
        "dbtableprefix": "oc_",             
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "logtimezone": "UTC",               
        "installed": true,                  
        "maintenance": false,               
        "theme": "",                        
        "loglevel": 2,            
        "mysql.utf8mb4": true,                         
        "memcache.local": "\\OC\\Memcache\\Redis",                                                                                                                                                                                                      
        "memcache.distributed": "\\OC\\Memcache\\Redis",      
        "memcache.locking": "\\OC\\Memcache\\Redis",    
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "port": 6379                              
        },                                              
        "mail_smtpmode": "smtp",                  
        "mail_smtpsecure": "ssl",
        "mail_sendmailmode": "smtp",
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpauthtype": "PLAIN",                    
        "mail_smtpauth": 1,                                                                                                 
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpport": "465",
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
        "updater.release.channel": "stable",
        "default_phone_region": "DE"
    }                                             
}

List of activated Apps

Enabled:         
  - activity: 2.19.0
  - bookmarks: 13.0.1            
  - bruteforcesettings: 2.7.0 
  - calendar: 4.4.4   
  - circles: 27.0.1
  - cloud_federation_api: 1.10.0
  - comments: 1.17.0     
  - contacts: 5.3.2
  - contactsinteraction: 1.8.0
  - cookbook: 0.10.2  
  - dashboard: 7.7.0      
  - dav: 1.27.0            
  - federatedfilesharing: 1.17.0 
  - federation: 1.17.0   
  - files: 1.22.0    
  - files_accesscontrol: 1.17.0
  - files: 1.22.0                                                                                                                                                                                                                                 [0/89]
  - files_accesscontrol: 1.17.0
  - files_markdown: 2.4.1    
  - files_pdfviewer: 2.8.0
  - files_rightclick: 1.6.0
  - files_sharing: 1.19.0     
  - files_texteditor: 2.15.1                      
  - files_trashbin: 1.17.0                              
  - files_versions: 1.20.0                          
  - firstrunwizard: 2.16.0
  - gpoddersync: 3.8.1                              
  - logreader: 2.12.0   
  - lookup_server_connector: 1.15.0
  - nextcloud_announcements: 1.16.0
  - notes: 4.8.1                 
  - notifications: 2.15.0           
  - oauth2: 1.15.1                                           
  - password_policy: 1.17.0                            
  - photos: 2.3.0                    
  - privacy: 1.11.0        
  - provisioning_api: 1.17.0                             
  - recommendations: 1.6.0     
  - related_resources: 1.2.0                             
  - serverinfo: 1.17.0                                       
  - settings: 1.9.0                         
  - sharebymail: 1.17.0             
  - support: 1.10.0
  - survey_client: 1.15.0
  - systemtags: 1.17.0                         
  - tasks: 0.15.0
  - theming: 2.2.0  
  - twofactor_backupcodes: 1.16.0
  - updatenotification: 1.17.0
  - user_status: 1.7.0
  - viewer: 2.1.0  
  - weather_status: 1.7.0       
  - workflowengine: 2.9.0
Disabled:          
  - admin_audit: 1.17.0       
  - encryption: 2.15.0
  - files_external: 1.19.0
  - suspicious_login: 5.0.0
  - text: 3.8.0 (installed 2.0.0)
  - twofactor_totp: 9.0.0
  - user_ldap: 1.17.0

Nextcloud Signing status

No errors have been found.

Nextcloud Logs

No response

Additional info

No response

@szaimen szaimen transferred this issue from nextcloud/server Aug 17, 2023
@juliushaertl
Copy link
Member

I cannot reproduce that, but haven't tried the community docker image yet. Any chance you can share a copy of such a file here as an attachment for checking the file encoding directly?

@code-surfer
Copy link
Author

code-surfer commented Aug 29, 2023

The file is fine, I thoroughly checked the encoding.
It is also displaying fine in the Nextcloud Android app.
The problem is in the dual pane Markdown editor that opens when I click the file in .../index.php/apps/files?dir=/Notizen/Medien&openfile=122800 which gets me to .../apps/files/?dir=/Notizen/Medien&fileid=88758#filestexteditor.

# cat vol/nextcloud/data/rj/files/Notizen/Medien/test.md
test äöü ⚰
# recode utf8..dump < vol/nextcloud/data/rj/files/Notizen/Medien/test.md
UCS2   Mne   Description

0074   t     latin small letter t
0065   e     latin small letter e
0073   s     latin small letter s
0074   t     latin small letter t
0020   SP    space
00E4   a:    latin small letter a with diaeresis
00F6   o:    latin small letter o with diaeresis
00FC   u:    latin small letter u with diaeresis
0020   SP    space
26B0      

Only when I edit the file and the editor saves it, then the encoding gets messed up. I removed the symbol that triggers the bug, but still:

# cat vol/nextcloud/data/rj/files/Notizen/Medien/test.md
test äöü
# recode utf8..dump < vol/nextcloud/data/rj/files/Notizen/Medien/test.md
UCS2   Mne   Description

0074   t     latin small letter t
0065   e     latin small letter e
0073   s     latin small letter s
0074   t     latin small letter t
0020   SP    space
00C3   A?    latin capital letter a with tilde
00A4   Cu    currency sign
00C3   A?    latin capital letter a with tilde
00B6   PI    pilcrow sign
00C3   A?    latin capital letter a with tilde
00BC   14    vulgar fraction one quarter

I restored the file.
Here is the base64-encoded content for your diagnostics:

# base64 vol/nextcloud/data/rj/files/Notizen/Medien/test.md
dGVzdCDDpMO2w7wg4pqwCg==

By the way, the bug is not triggered by every character beyond U+00FF. With the € U+20AC it works fine. Same with ✄ U+2704. But with ⚰ U+26B0 it breaks.

Code point Result
20AC fine
26A5 broken
26AF broken
26B0 broken
26B1 broken
26B2 broken
26FF fine
2704 fine

@juliushaertl
Copy link
Member

Ah, you are using a different app then, let me move the issue

@juliushaertl juliushaertl transferred this issue from nextcloud/text Aug 29, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants