Browse json files exported by Tyrrrz/DiscordChatExporter in familiar discord like user interface
Home Page: https://dcef.slada.sk
License: GNU General Public License v3.0
sorry but i dont understand how i can install it on linux would be nice if i can get some help
2022/12/30 10:14:07 [emerg] 11632#3000: unknown "discord" variable 2022/12/30 10:14:32 [emerg] 7976#5340: unknown "discord" variable 2022/12/30 10:15:48 [emerg] 11480#6832: unknown "discord" variable 2022/12/30 10:18:49 [emerg] 9884#9888: unknown "discord" variable 2022/12/30 10:19:07 [emerg] 3364#3368: unknown "discord" variable 2022/12/30 10:19:58 [emerg] 3020#8440: unknown "discord" variable
This is what the error.log contains inside the nginx>logs
I tried disabling the antivirus and also the older version, but I always get a blank page with microsoft edge:
Thank you for the effort creating such tool, I would be vare greateful if you can help me.
I edited this issue to remove cryptocurrency adspam. My account was used by a bad actor because of a leaked access token. This bad actor created lots of issues with cryptocurrency adspam. I am sorry.
Add some option to allow securing the server behind a basic auth user/password.
INFO: 192.168.1.28:0 - "GET /guilds HTTP/1.1" 500 Internal Server Error
ERROR: Exception in ASGI application
Traceback (most recent call last):
File "/usr/local/lib/python3.11/dist-packages/uvicorn/protocols/http/httptools_impl.py", line 426, in run_asgi
result = await app( # type: ignore[func-returns-value]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/uvicorn/middleware/proxy_headers.py", line 84, in call
return await self.app(scope, receive, send)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/fastapi/applications.py", line 270, in call
await super().call(scope, receive, send)
File "/usr/local/lib/python3.11/dist-packages/starlette/applications.py", line 124, in call
await self.middleware_stack(scope, receive, send)
File "/usr/local/lib/python3.11/dist-packages/starlette/middleware/errors.py", line 184, in call
raise exc
File "/usr/local/lib/python3.11/dist-packages/starlette/middleware/errors.py", line 162, in call
await self.app(scope, receive, _send)
File "/usr/local/lib/python3.11/dist-packages/starlette/middleware/exceptions.py", line 79, in call
raise exc
File "/usr/local/lib/python3.11/dist-packages/starlette/middleware/exceptions.py", line 68, in call
await self.app(scope, receive, sender)
File "/usr/local/lib/python3.11/dist-packages/fastapi/middleware/asyncexitstack.py", line 21, in call
raise e
File "/usr/local/lib/python3.11/dist-packages/fastapi/middleware/asyncexitstack.py", line 18, in call
await self.app(scope, receive, send)
File "/usr/local/lib/python3.11/dist-packages/starlette/routing.py", line 706, in call
await route.handle(scope, receive, send)
File "/usr/local/lib/python3.11/dist-packages/starlette/routing.py", line 276, in handle
await self.app(scope, receive, send)
File "/usr/local/lib/python3.11/dist-packages/starlette/routing.py", line 66, in app
response = await func(request)
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/fastapi/routing.py", line 237, in app
raw_response = await run_endpoint_function(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/fastapi/routing.py", line 163, in run_endpoint_function
return await dependant.call(**values)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/dcef/backend/fastapi/app.py", line 103, in get_guilds
return list(cursor)
^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/pymongo/cursor.py", line 1248, in next
if len(self.__data) or self._refresh():
^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/pymongo/cursor.py", line 1139, in _refresh
self.__session = self.__collection.database.client._ensure_session()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/pymongo/mongo_client.py", line 1740, in _ensure_session
return self.__start_session(True, causal_consistency=False)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/pymongo/mongo_client.py", line 1685, in __start_session
self._topology._check_implicit_session_support()
File "/usr/local/lib/python3.11/dist-packages/pymongo/topology.py", line 538, in _check_implicit_session_support
self._check_session_support()
File "/usr/local/lib/python3.11/dist-packages/pymongo/topology.py", line 554, in _check_session_support
self._select_servers_loop(
File "/usr/local/lib/python3.11/dist-packages/pymongo/topology.py", line 238, in _select_servers_loop
raise ServerSelectionTimeoutError(
pymongo.errors.ServerSelectionTimeoutError: 127.0.0.1:27017: [Errno 111] Connection refused, Timeout: 30s, Topology Description: <TopologyDescription id: 6500478f77df2a317b18a426, topology_type: Unknown, servers: [<ServerDescription ('127.0.0.1', 27017) server_type: Unknown, rtt: None, error=AutoReconnect('127.0.0.1:27017: [Errno 111] Connection refused')>]>
2023/09/12 12:01:50 [error] 21#21: *18 open() "/dcef/cache/preprocess/fonts/ggsans-normal-400.woff2" failed (2: No such file or directory), client: 192.168.1.28, server: localhost, request: "GET /data/fonts/ggsans-normal-400.woff2 HTTP/1.1", host: "192.168.1.39:21011", referrer: "http://192.168.1.39:21011/_app/immutable/assets/_layout-92ac35d3.css"
2023/09/12 12:01:50 [error] 21#21: *19 open() "/dcef/cache/preprocess/fonts/ggsans-normal-700.woff2" failed (2: No such file or directory), client: 192.168.1.28, server: localhost, request: "GET /data/fonts/ggsans-normal-700.woff2 HTTP/1.1", host: "192.168.1.39:21011", referrer: "http://192.168.1.39:21011/_app/immutable/assets/_layout-92ac35d3.css"
I can’t find any documentation for using this program without docker on Linux. How can I accomplish that and if there currently is not a way to do so could that be changed?
To see something in the DCEF window.
DCEF window is just solid white.
--version): v2.39.1How did you export the data?
Is there any relevant information in logs? If so, please provide them:
dcef/logs.txt: logs.txt
The backend error log was empty.
I don't really know what the problem could be. I read through the logs; nothing stuck out to me.
Hi there,
I'm trying to export messages from this particular channel and I'm getting this message:
here's the channel in question when I view it in Discord:
I tried to look up the guides but couldn't find much on this. Could someone point me in the right direction please?
Just a nitpick I noticed.
On Discord, when a message contains only emoji and no text, those emoji are larger than average. DCEF (at least the version 1.13.2 that I'm using) doesn't do this, instead displaying them at the same size they would be if there was text in the message.
Hopefully I'm not reporting something that's already been fixed in later versions. I couldn't find any issues about this, at least.
Dear programmer, your project is so professional and everthing included I love it.
But unfortunatelly the front-end project didnt work fully.
So I installed: DiscordChatExporter-frontend-v1.13.0-win.zip
And then I extracted it.
And I put JSON export of the DiscordChatExporter made by GUI in the exports directory.
But I cant see anything still.
Error: https://prnt.sc/yW4dex1TBZqU
Exports included: https://prnt.sc/CaBZOB2VAfsu
Any reason why and how to fix?
I see Linux support was shown in #4. I'm curious if this will run on a M1 MacBook (Apple Silicon)?
Originally posted by Oguret2 December 3, 2023
Attached files are not displayed. There are no problems with attached photos or videos, but other files are not displayed at all. There are files in the attachment folder and they are listed in the chat json. Add the ability to display files of any format plz.
It would be very useful if the program could read compressed exports. It would save storage space and the user wont have to decompress compressed exports in order to use them.
Markdown should look similar to how it displays on Discord - quotes prefixed with a single quote mark should only quote one line. The entire message should be quoted only with triple quote marks, or a quote mark on each line.
This message (on Discord):
Renders as such (on DCEF):
There is also additional whitespace between the name and the start of the quote.
--version): v2.39.1How did you export the data?
--format): Json--markdown): NoThe message content from the above screenshots, from "print message object to devtools":
"content": "> would require people to open worlds in the newest version and hit /save\nI think that works tbh"
Sometimes search doesn't jump to message correctly. Please check the video: https://streamable.com/bh1ola
File detected:
\DiscordChatExporter-frontend-v2.9.0-win\dcef\backend\http-server\http-server.exe
Version:
DiscordChatExporter-frontend-v2.9.0-win
Proof / Info:
https://tip.neiki.dev/file/823cf5738baff045fa71783fd1f2f5f7656f0e1d99518ccb57c479824eaef763
https://www.virustotal.com/gui/file/823cf5738baff045fa71783fd1f2f5f7656f0e1d99518ccb57c479824eaef763
https://opentip.kaspersky.com/823cf5738baff045fa71783fd1f2f5f7656f0e1d99518ccb57c479824eaef763/results
I want to make this software Japanese, but I don't know how to do it. Please tell me where I should edit!
Application starts
...
See logs and notes
...
--version): v2.42.2How did you export the data?
--format): Json--media): Yes (and --reuse-media_--markdown): set to falseIs there any relevant information in logs? If so, please provide them:
2023-11-05 04:49:44.915173+00:00[Preprocess] took 0.64 seconds
2023-11-05 04:49:44.915217+00:00Traceback (most recent call last):
2023-11-05 04:49:44.915228+00:00File "/dcef/backend/preprocess/main_mongo.py", line 146, in <module>
2023-11-05 04:49:44.915368+00:00main(input_dir, output_dir)
2023-11-05 04:49:44.915413+00:00File "/dcef/backend/preprocess/main_mongo.py", line 119, in main
2023-11-05 04:49:44.915485+00:00channel_cache = ChannelCache()
2023-11-05 04:49:44.915507+00:00^^^^^^^^^^^^^^
2023-11-05 04:49:44.915521+00:00File "/dcef/backend/preprocess/ChannelCache.py", line 22, in __init__
2023-11-05 04:49:44.915568+00:00os.makedirs(self.cache_folder_path)
2023-11-05 04:49:44.915594+00:00File "<frozen os>", line 215, in makedirs
2023-11-05 04:49:44.915605+00:00File "<frozen os>", line 215, in makedirs
2023-11-05 04:49:44.915617+00:00File "<frozen os>", line 215, in makedirs
2023-11-05 04:49:44.915624+00:00[Previous line repeated 1 more time]
2023-11-05 04:49:44.915630+00:00File "<frozen os>", line 225, in makedirs
2023-11-05 04:49:44.915636+00:00OSError: [Errno 30] Read-only file system: '../../release'
Notes
This is actually running on a TrueNAS SCALE system, which is a debian linux appliance that runs a kubernetes cluster. I've ran into a few weird readonly errors prior to this but was able to resolve them. After I saw the most recent commit was to fix the docker image, I also made sure to pull the image by sha256 digest to make sure it was the most up to date.
/dcef/cache -> PVC Volume
/dcef/exports -> points to my DCE exports directory. Exports were generated with incremental chat exporter.
/var/lib/nginx -> PVC Volume (without this, /var/lib/nginx/body can't be written, NGINX throws an EMERG error and the container quits)
/var/log/nginx -> PVC Volume (without this, being unable to write access.log triggers an EMERG error and container quits)
The container is running as uid/gid 0/0 (root) as without this an error is thrown. Ports are mapped 21011 to 21011, which then feeds into a kubernetes ingress controller. Access works fine, webpage loads fine, it's only loading the guilds that fails.
I know this is probably beyond what was intended, so if there's anyway I can help let me know.
I don't actually know if it's intended to run on Windows 7 at all, but I didn't see any OS version requirements listed anywhere.
I'm also just generally curious if it's possible for it to run on Windows 7 with some additional user-side bandaids, even if it's not explicitly intended for it.
When I initially started the program, it complained about a missing api-ms-win-core-path-l1-1-0.dll, which I added to my system manually. After that, when I attempt to start it, Command Prompt opens and immediately closes with the message
Error loading Python DLL: 'C:\Users\USERNAME\AppData\Local\Temp_MEI123456\python311.dll'.
No error messages generated.
FormatMessageW: The parameter is incorrect.
LoadLibrary: PyInstaller: FormatMessageW failed.
The mentioned AppData folder always has a different numerical identifier, and none of the folders exist, although I notice similar folders do exist in the AppData folder of the Windows 10 PC where I run DCEF.
The program would run.
The program does not run.
N/A: Tested with just the default guild.
Is there any relevant information in logs? If so, please provide them:
No logs were generated at all.
When looking for local paths, glob ignores hidden files (eg .foobar-ABCDE) and the regex ignores empty filenames (ABCDE).
The first issue can be fixed by using the include_hidden glob keyword that was added in Python 3.11. However this would cause older Python versions to error, so the TypeError exception would need to be caught with a fallback that doesn't use the keyword.
I'm not sure what the best way to solve the empty filenames situation would be, possibly making the strip_hash_from_filename method return a generic string.
In practice, the hidden filenames sometimes occur when images are uploaded to discord and look something like .eJwNxVEOgyAMANC7cAAKZQLxNgQJmmlLaM0-lt19v-4EED2.pqslNVLVEiec8IxkctgeUP5kocc. The empty filenames sometimes occur for URL thumbnails.
DCEF starts faster, doesn't get hung up on specific steps as detailed below
DCEF starts and outputs messages like 2023-07-16 22:52:09.447194 mongodb: {"t":{"$date":"2023-07-16T22:52:09.447-04:00"},"s":"I", "c":"EXECUTOR", "id":6983000, "ctx":"conn1","msg":"Slow SessionWorkflow loop","attr":{"elapsed":{"totalMillis":10000,"activeMillis":10000,"receiveWorkMillis":0,"processWorkMillis":10000,"sendResponseMillis":0,"finalizeMillis":0}}} when processing JSONs. This could be a problem with how the JSONs got exported or something, I can provide some of them if need be.
Ryzen 5 5600
GTX 1080 ti
32 GB RAM
--version): v2.40.0How did you export the data?
--format): Json--media): No--markdown): No (or at least, I don't specify --markdown)Is there any relevant information in logs? If so, please provide them:
Here's a snippet showing the messages, and how long they persist.
https://pastebin.com/hDh12K7v
I'm basically mirroring a report I made on the Discord Chat Exporter Github because I don't know of any way to tell whether this is an issue with the export/data or with the rendering/parsing of the data. Feel free to close this if you know it's out of your hands.
All emoji display successfully.
Emoji which are from other servers (besides the one which was exported) and are posted in-line with the message text rather than as reactions, seem to very frequently show up in chat as :emojiname: rather than the image of the actual emoji.
Sometimes such emoji will display fine, no idea why. Sometimes one of them will display fine and an emoji right below it that's from the exported server will fail to render. I've seen at least one instance of an emoji from another server rendering, but showing the wrong emoji when compared side-by-side with the live channel from which it was extracted.
--version): v2.39.1How did you export the data?
--format): Json--media): Yes--markdown): Yeswhenever i try to open the start_viewer.bat it shows that 127.0.0.1 error in a browser window and
C:\Users\xyz\Downloads\DiscordChatExporter-frontend-v1.8.1-win\bin>if exist "C:\Program Files\Google\Chrome\Application\chrome.exe" ("C:\Program Files\Google\Chrome\Application\chrome.exe" --app="http://127.0.0.1:21011/index.html" --enable-precise-memory-info ) else (rundll32 url.dll,FileProtocolHandler http://127.0.0.1:21011/index.html )
this error. help???
Currently it appears the channels and categories are sorted by amount of messages, however this behavior limits one's abilities to accurately recreate servers within the software; a feature which would massively helpful for those seeking to archive these servers as they appear in Discord. A clunky Config file, or similar would suffice, I don't believe that anything fancy has to be implemented but simply having the option would really help improve the usability of the software for this use case.
Either way thank you in advance!
I'm getting this error when running the container:
$ docker run --volume "$(pwd):/dcef/exports" --volume dcef_cache:/dcef/cache --rm -p 21011:21011 -it dcef
Running uncompiled version of preprocess.py
Found 24 JSON files
Found 4 HTML files
Found 2929 media files
Hash: 165f7eda972a09a2ec21e550665c76283ab113d340d02d1b431654836c757b10
Hash is different, cache is not up to date
Sorting JSONs by guild_id...
Traceback (most recent call last):
File "/dcef/backend/preprocess/main.py", line 29, in <module>
main(input_dir, output_dir)
File "/dcef/backend/preprocess/main.py", line 15, in main
p.process()
File "/dcef/backend/preprocess/Preprocess.py", line 254, in process
guild_id = helpers.pad_id(data['guild']['id'])
~~~~~~~~~~~~~^^^^^^
TypeError: 'int' object is not subscriptable
Already starts with mongodb not supported (even tho I don't understand why you would need a database do display local files).
Here are the logs if needed, as you have written you are not sure if its compatible. I just wanted to provide some information. You can close the issue ^^
Last login: Sun Jul 23 13:36:04 on ttys003
MacBook-Air-von-Kevin Backup % docker run --volume "$(pwd):/dcef/exports" --volume dcef_cache:/dcef/cache --rm --name dcef -p 21011:21011 -it slada/dcef:main
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
WARNING: MongoDB 5.0+ requires a CPU with AVX support, and your current system does not appear to have that!
see https://jira.mongodb.org/browse/SERVER-54407
see also https://www.mongodb.com/community/forums/t/mongodb-5-0-cpu-intel-g4650-compatibility/116610/2
see also docker-library/mongo#485 (comment)
main_mongo loaded
Connecting to database...
Traceback (most recent call last):
File "/dcef/backend/preprocess/main_mongo.py", line 80, in
main(input_dir, output_dir)
File "/dcef/backend/preprocess/main_mongo.py", line 50, in main
database = MongoDatabase()
^^^^^^^^^^^^^^^
File "/dcef/backend/preprocess/MongoDatabase.py", line 27, in init
self.create_indexes()
File "/dcef/backend/preprocess/MongoDatabase.py", line 32, in create_indexes
self.col["messages"].create_index("channelId", default_language="none")
File "/usr/local/lib/python3.11/dist-packages/pymongo/collection.py", line 2055, in create_index
return self.__create_indexes([index], session, **cmd_options)[0]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/pymongo/_csot.py", line 105, in csot_wrapper
return func(self, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/pymongo/collection.py", line 1911, in __create_indexes
with self._socket_for_writes(session) as sock_info:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/pymongo/collection.py", line 248, in _socket_for_writes
return self.__database.client._socket_for_writes(session)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/pymongo/mongo_client.py", line 1267, in _socket_for_writes
server = self._select_server(writable_server_selector, session)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/pymongo/mongo_client.py", line 1257, in _select_server
server = topology.select_server(server_selector)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/pymongo/topology.py", line 272, in select_server
server = self._select_server(selector, server_selection_timeout, address)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/pymongo/topology.py", line 261, in _select_server
servers = self.select_servers(selector, server_selection_timeout, address)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/pymongo/topology.py", line 223, in select_servers
server_descriptions = self._select_servers_loop(selector, server_timeout, address)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/pymongo/topology.py", line 238, in _select_servers_loop
raise ServerSelectionTimeoutError(
pymongo.errors.ServerSelectionTimeoutError: 127.0.0.1:27017: [Errno 111] Connection refused, Timeout: 30s, Topology Description: <TopologyDescription id: 64bd10cd1634855f63a3b7d4, topology_type: Unknown, servers: [<ServerDescription ('127.0.0.1', 27017) server_type: Unknown, rtt: None, error=AutoReconnect('127.0.0.1:27017: [Errno 111] Connection refused')>]>
############################################################
############################################################
2023/07/23 11:37:16 [notice] 38#38: using the "epoll" event method
2023/07/23 11:37:16 [notice] 38#38: nginx/1.18.0 (Ubuntu)
2023/07/23 11:37:16 [notice] 38#38: OS: Linux 5.15.49-linuxkit
2023/07/23 11:37:16 [notice] 38#38: getrlimit(RLIMIT_NOFILE): 1048576:1048576
2023/07/23 11:37:16 [notice] 38#38: start worker processes
2023/07/23 11:37:16 [notice] 38#38: start worker process 41
INFO: Started server process [35]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://0.0.0.0:58000 (Press CTRL+C to quit)
INFO: 172.17.0.1:0 - "GET /guilds HTTP/1.1" 500 Internal Server Error
ERROR: Exception in ASGI application
Traceback (most recent call last):
File "/usr/local/lib/python3.11/dist-packages/uvicorn/protocols/http/httptools_impl.py", line 426, in run_asgi
result = await app( # type: ignore[func-returns-value]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/uvicorn/middleware/proxy_headers.py", line 84, in call
return await self.app(scope, receive, send)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/fastapi/applications.py", line 270, in call
await super().call(scope, receive, send)
File "/usr/local/lib/python3.11/dist-packages/starlette/applications.py", line 124, in call
await self.middleware_stack(scope, receive, send)
File "/usr/local/lib/python3.11/dist-packages/starlette/middleware/errors.py", line 184, in call
raise exc
File "/usr/local/lib/python3.11/dist-packages/starlette/middleware/errors.py", line 162, in call
await self.app(scope, receive, _send)
File "/usr/local/lib/python3.11/dist-packages/starlette/middleware/exceptions.py", line 79, in call
raise exc
File "/usr/local/lib/python3.11/dist-packages/starlette/middleware/exceptions.py", line 68, in call
await self.app(scope, receive, sender)
File "/usr/local/lib/python3.11/dist-packages/fastapi/middleware/asyncexitstack.py", line 21, in call
raise e
File "/usr/local/lib/python3.11/dist-packages/fastapi/middleware/asyncexitstack.py", line 18, in call
await self.app(scope, receive, send)
File "/usr/local/lib/python3.11/dist-packages/starlette/routing.py", line 706, in call
await route.handle(scope, receive, send)
File "/usr/local/lib/python3.11/dist-packages/starlette/routing.py", line 276, in handle
await self.app(scope, receive, send)
File "/usr/local/lib/python3.11/dist-packages/starlette/routing.py", line 66, in app
response = await func(request)
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/fastapi/routing.py", line 237, in app
raw_response = await run_endpoint_function(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/fastapi/routing.py", line 163, in run_endpoint_function
return await dependant.call(**values)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/dcef/backend/fastapi/app.py", line 102, in get_guilds
return list(cursor)
^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/pymongo/cursor.py", line 1248, in next
if len(self.__data) or self._refresh():
^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/pymongo/cursor.py", line 1139, in _refresh
self.__session = self.__collection.database.client._ensure_session()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/pymongo/mongo_client.py", line 1740, in _ensure_session
return self.__start_session(True, causal_consistency=False)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/pymongo/mongo_client.py", line 1685, in __start_session
self._topology._check_implicit_session_support()
File "/usr/local/lib/python3.11/dist-packages/pymongo/topology.py", line 538, in _check_implicit_session_support
self._check_session_support()
File "/usr/local/lib/python3.11/dist-packages/pymongo/topology.py", line 554, in _check_session_support
self._select_servers_loop(
File "/usr/local/lib/python3.11/dist-packages/pymongo/topology.py", line 238, in _select_servers_loop
raise ServerSelectionTimeoutError(
pymongo.errors.ServerSelectionTimeoutError: 127.0.0.1:27017: [Errno 111] Connection refused, Timeout: 30s, Topology Description: <TopologyDescription id: 64bd10ee492cb8404f48613b, topology_type: Unknown, servers: [<ServerDescription ('127.0.0.1', 27017) server_type: Unknown, rtt: None, error=AutoReconnect('127.0.0.1:27017: [Errno 111] Connection refused')>]>
INFO: 172.17.0.1:0 - "GET /guilds HTTP/1.1" 500 Internal Server Error
ERROR: Exception in ASGI application
Traceback (most recent call last):
File "/usr/local/lib/python3.11/dist-packages/uvicorn/protocols/http/httptools_impl.py", line 426, in run_asgi
result = await app( # type: ignore[func-returns-value]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/uvicorn/middleware/proxy_headers.py", line 84, in call
return await self.app(scope, receive, send)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/fastapi/applications.py", line 270, in call
await super().call(scope, receive, send)
File "/usr/local/lib/python3.11/dist-packages/starlette/applications.py", line 124, in call
await self.middleware_stack(scope, receive, send)
File "/usr/local/lib/python3.11/dist-packages/starlette/middleware/errors.py", line 184, in call
raise exc
File "/usr/local/lib/python3.11/dist-packages/starlette/middleware/errors.py", line 162, in call
await self.app(scope, receive, _send)
File "/usr/local/lib/python3.11/dist-packages/starlette/middleware/exceptions.py", line 79, in call
raise exc
File "/usr/local/lib/python3.11/dist-packages/starlette/middleware/exceptions.py", line 68, in call
await self.app(scope, receive, sender)
File "/usr/local/lib/python3.11/dist-packages/fastapi/middleware/asyncexitstack.py", line 21, in call
raise e
File "/usr/local/lib/python3.11/dist-packages/fastapi/middleware/asyncexitstack.py", line 18, in call
await self.app(scope, receive, send)
File "/usr/local/lib/python3.11/dist-packages/starlette/routing.py", line 706, in call
await route.handle(scope, receive, send)
File "/usr/local/lib/python3.11/dist-packages/starlette/routing.py", line 276, in handle
await self.app(scope, receive, send)
File "/usr/local/lib/python3.11/dist-packages/starlette/routing.py", line 66, in app
response = await func(request)
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/fastapi/routing.py", line 237, in app
raw_response = await run_endpoint_function(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/fastapi/routing.py", line 163, in run_endpoint_function
return await dependant.call(**values)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/dcef/backend/fastapi/app.py", line 102, in get_guilds
return list(cursor)
^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/pymongo/cursor.py", line 1248, in next
if len(self.__data) or self._refresh():
^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/pymongo/cursor.py", line 1139, in _refresh
self.__session = self.__collection.database.client._ensure_session()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/pymongo/mongo_client.py", line 1740, in _ensure_session
return self.__start_session(True, causal_consistency=False)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/pymongo/mongo_client.py", line 1685, in __start_session
self._topology._check_implicit_session_support()
File "/usr/local/lib/python3.11/dist-packages/pymongo/topology.py", line 538, in _check_implicit_session_support
self._check_session_support()
File "/usr/local/lib/python3.11/dist-packages/pymongo/topology.py", line 554, in _check_session_support
self._select_servers_loop(
File "/usr/local/lib/python3.11/dist-packages/pymongo/topology.py", line 238, in _select_servers_loop
raise ServerSelectionTimeoutError(
pymongo.errors.ServerSelectionTimeoutError: 127.0.0.1:27017: [Errno 111] Connection refused, Timeout: 30s, Topology Description: <TopologyDescription id: 64bd10ee492cb8404f48613b, topology_type: Unknown, servers: [<ServerDescription ('127.0.0.1', 27017) server_type: Unknown, rtt: None, error=AutoReconnect('127.0.0.1:27017: [Errno 111] Connection refused')>]>
2023/07/23 11:38:40 [error] 41#41: *8 open() "/dcef/cache/preprocess/fonts/ggsans-normal-400.woff2" failed (2: No such file or directory), client: 172.17.0.1, server: localhost, request: "GET /data/fonts/ggsans-normal-400.woff2 HTTP/1.1", host: "127.0.0.1:21011", referrer: "http://127.0.0.1:21011/_app/immutable/assets/_layout-2d954d72.css"
2023/07/23 11:38:40 [error] 41#41: *9 open() "/dcef/cache/preprocess/fonts/ggsans-normal-700.woff2" failed (2: No such file or directory), client: 172.17.0.1, server: localhost, request: "GET /data/fonts/ggsans-normal-700.woff2 HTTP/1.1", host: "127.0.0.1:21011", referrer: "http://127.0.0.1:21011/_app/immutable/assets/_layout-2d954d72.css"
2023/07/23 11:38:56 [error] 41#41: *8 open() "/dcef/cache/preprocess/fonts/ggsans-normal-600.woff2" failed (2: No such file or directory), client: 172.17.0.1, server: localhost, request: "GET /data/fonts/ggsans-normal-600.woff2 HTTP/1.1", host: "127.0.0.1:21011", referrer: "http://127.0.0.1:21011/_app/immutable/assets/_layout-2d954d72.css"
2023/07/23 11:38:56 [error] 41#41: *9 open() "/dcef/cache/preprocess/fonts/ggsans-normal-500.woff2" failed (2: No such file or directory), client: 172.17.0.1, server: localhost, request: "GET /data/fonts/ggsans-normal-500.woff2 HTTP/1.1", host: "127.0.0.1:21011", referrer: "http://127.0.0.1:21011/_app/immutable/assets/_layout-2d954d72.css"
2023/07/23 11:39:03 [error] 41#41: *6 open() "/dcef/cache/preprocess/fonts/ggsans-normal-400.woff2" failed (2: No such file or directory), client: 172.17.0.1, server: localhost, request: "GET /data/fonts/ggsans-normal-400.woff2 HTTP/1.1", host: "localhost:21011", referrer: "http://localhost:21011/_app/immutable/assets/_layout-2d954d72.css"
2023/07/23 11:39:03 [error] 41#41: *15 open() "/dcef/cache/preprocess/fonts/ggsans-normal-700.woff2" failed (2: No such file or directory), client: 172.17.0.1, server: localhost, request: "GET /data/fonts/ggsans-normal-700.woff2 HTTP/1.1", host: "localhost:21011", referrer: "http://localhost:21011/_app/immutable/assets/_layout-2d954d72.css"
2023/07/23 11:39:03 [error] 41#41: *16 open() "/dcef/cache/preprocess/fonts/ggsans-normal-500.woff2" failed (2: No such file or directory), client: 172.17.0.1, server: localhost, request: "GET /data/fonts/ggsans-normal-500.woff2 HTTP/1.1", host: "localhost:21011", referrer: "http://localhost:21011/_app/immutable/assets/_layout-2d954d72.css"
2023/07/23 11:39:03 [error] 41#41: *17 open() "/dcef/cache/preprocess/fonts/ggsans-normal-600.woff2" failed (2: No such file or directory), client: 172.17.0.1, server: localhost, request: "GET /data/fonts/ggsans-normal-600.woff2 HTTP/1.1", host: "localhost:21011", referrer: "http://localhost:21011/_app/immutable/assets/_layout-2d954d72.css"
2023/07/23 11:40:38 [notice] 41#41: signal 28 (SIGWINCH) received
2023/07/23 11:40:38 [notice] 38#38: signal 28 (SIGWINCH) received
2023/07/23 11:40:38 [info] 41#41: epoll_wait() failed (4: Interrupted system call)
2023/07/23 11:40:38 [notice] 38#38: signal 28 (SIGWINCH) received
2023/07/23 11:40:38 [notice] 41#41: signal 28 (SIGWINCH) received
2023/07/23 11:40:38 [info] 41#41: epoll_wait() failed (4: Interrupted system call)
2023/07/23 11:40:38 [notice] 38#38: signal 28 (SIGWINCH) received
2023/07/23 11:40:38 [notice] 41#41: signal 28 (SIGWINCH) received
2023/07/23 11:40:38 [info] 41#41: epoll_wait() failed (4: Interrupted system call)
2023/07/23 11:40:38 [notice] 38#38: signal 28 (SIGWINCH) received
2023/07/23 11:40:38 [notice] 41#41: signal 28 (SIGWINCH) received
2023/07/23 11:40:38 [info] 41#41: epoll_wait() failed (4: Interrupted system call)
2023/07/23 11:40:38 [notice] 38#38: signal 28 (SIGWINCH) received
2023/07/23 11:40:38 [notice] 41#41: signal 28 (SIGWINCH) received
2023/07/23 11:40:38 [info] 41#41: epoll_wait() failed (4: Interrupted system call)
2023/07/23 11:40:38 [notice] 41#41: signal 28 (SIGWINCH) received
The program gets detected as a trojan by Windows Defender.
It says this was detected:
Trojan:Win32/Bearfoos.A!ml
This is more of a question than a report.
On the old browser-based version (v1.13.2) is there any way to view very large exports (on the order of 2GB-3GB just for the JSON file)?
When I even mouse over the guild, Chrome kills the tab with an OOM error once it hits 1GB usage, and while Firefox actually does reach around 2.5GB usage, the page never opens and the memory usage quickly goes back to normal. Even if I navigate to the guild in Firefox via the URL, the page just remains solid white.
I assume the newer, non-browser-based version can handle this natively (as per #16) but it unfortunately won't run properly for me, so I'd like to know if it's possible to view such large exports even on the old version.
Do I need to use a partitioned export instead?
Support for https://github.com/chylex/Discord-History-Tracker would be nice as it supports incremental backups
I can access the program from the web interface and it works
This has the same behavior as the one described in #32 .
Accessing the web interface gives me a fully white page.
Mongo complains about the CPU not having AVX support (see logs)
I am using the following compose.yml
version: '3.9'
services:
dcef:
image: 'slada/dcef:main'
ports:
- '21011:21011'
container_name: dcef
volumes:
- 'dcef_cache:/dcef/cache'
- './exports:/dcef/exports'
restart: unless-stopped
volumes:
dcef_cache:docker pull slada/dcef:mainsudo docker compose up -d --force-recreate--version): v2.42.0How did you export the data?
--format): Json--media): Yes--markdown): No--dateformat dd/MM/yyyy HH:mm:ss, --reuse-mediaIs there any relevant information in logs? If so, please provide them:
WARNING: MongoDB 5.0+ requires a CPU with AVX support, and your current system does not appear to have that!
see https://jira.mongodb.org/browse/SERVER-54407
see also https://www.mongodb.com/community/forums/t/mongodb-5-0-cpu-intel-g4650-compatibility/116610/2
see also https://github.com/docker-library/mongo/issues/485#issuecomment-891991814
main_mongo loaded
Connecting to database...
[Preprocess] took 30.1 seconds
Traceback (most recent call last):
File "/dcef/backend/preprocess/main_mongo.py", line 145, in <module>
main(input_dir, output_dir)
File "/dcef/backend/preprocess/main_mongo.py", line 100, in main
wipe_database(database)
File "/dcef/backend/preprocess/main_mongo.py", line 32, in wipe_database
whitelisted_guild_ids = config.find_one({"key": "whitelisted_guild_ids"})
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/pymongo/collection.py", line 1459, in find_one
for result in cursor.limit(-1):
File "/usr/local/lib/python3.11/dist-packages/pymongo/cursor.py", line 1248, in next
if len(self.__data) or self._refresh():
^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/pymongo/cursor.py", line 1139, in _refresh
self.__session = self.__collection.database.client._ensure_session()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/pymongo/mongo_client.py", line 1740, in _ensure_session
return self.__start_session(True, causal_consistency=False)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/pymongo/mongo_client.py", line 1685, in __start_session
self._topology._check_implicit_session_support()
File "/usr/local/lib/python3.11/dist-packages/pymongo/topology.py", line 538, in _check_implicit_session_support
self._check_session_support()
File "/usr/local/lib/python3.11/dist-packages/pymongo/topology.py", line 554, in _check_session_support
self._select_servers_loop(
File "/usr/local/lib/python3.11/dist-packages/pymongo/topology.py", line 238, in _select_servers_loop
raise ServerSelectionTimeoutError(
pymongo.errors.ServerSelectionTimeoutError: 127.0.0.1:27017: [Errno 111] Connection refused, Timeout: 30s, Topology Description: <TopologyDescription id: 653e9b253a02030314937ff4, topology_type: Unknown, servers: [<ServerDescription ('127.0.0.1', 27017) server_type: Unknown, rtt: None, error=AutoReconnect('127.0.0.1:27017: [Errno 111] Connection refused')>]>
/dcef/run_container.sh: line 6: 13 Illegal instruction (core dumped) mongod --dbpath "/dcef/cache/db/" --wiredTigerCacheSizeGB 1.5 (wd: /dcef)
############################################################
# Open http://127.0.0.1:21011/ in your browser to view GUI #
############################################################
2023/10/29 17:49:56 [notice] 18#18: using the "epoll" event method
2023/10/29 17:49:56 [notice] 18#18: nginx/1.18.0 (Ubuntu)
2023/10/29 17:49:56 [notice] 18#18: OS: Linux 5.15.131-1-MANJARO
2023/10/29 17:49:56 [notice] 18#18: getrlimit(RLIMIT_NOFILE): 1073741816:1073741816
2023/10/29 17:49:56 [notice] 18#18: start worker processes
2023/10/29 17:49:56 [notice] 18#18: start worker process 19
INFO: Started server process [17]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://0.0.0.0:58000 (Press CTRL+C to quit)
INFO: [CENSORED]:0 - "GET /guilds HTTP/1.1" 500 Internal Server Error
ERROR: Exception in ASGI application
Traceback (most recent call last):
File "/usr/local/lib/python3.11/dist-packages/uvicorn/protocols/http/httptools_impl.py", line 426, in run_asgi
result = await app( # type: ignore[func-returns-value]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/uvicorn/middleware/proxy_headers.py", line 84, in __call__
return await self.app(scope, receive, send)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/fastapi/applications.py", line 270, in __call__
await super().__call__(scope, receive, send)
File "/usr/local/lib/python3.11/dist-packages/starlette/applications.py", line 124, in __call__
await self.middleware_stack(scope, receive, send)
File "/usr/local/lib/python3.11/dist-packages/starlette/middleware/errors.py", line 184, in __call__
raise exc
File "/usr/local/lib/python3.11/dist-packages/starlette/middleware/errors.py", line 162, in __call__
await self.app(scope, receive, _send)
File "/usr/local/lib/python3.11/dist-packages/starlette/middleware/exceptions.py", line 79, in __call__
raise exc
File "/usr/local/lib/python3.11/dist-packages/starlette/middleware/exceptions.py", line 68, in __call__
await self.app(scope, receive, sender)
File "/usr/local/lib/python3.11/dist-packages/fastapi/middleware/asyncexitstack.py", line 21, in __call__
raise e
File "/usr/local/lib/python3.11/dist-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
await self.app(scope, receive, send)
File "/usr/local/lib/python3.11/dist-packages/starlette/routing.py", line 706, in __call__
await route.handle(scope, receive, send)
File "/usr/local/lib/python3.11/dist-packages/starlette/routing.py", line 276, in handle
await self.app(scope, receive, send)
File "/usr/local/lib/python3.11/dist-packages/starlette/routing.py", line 66, in app
response = await func(request)
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/fastapi/routing.py", line 237, in app
raw_response = await run_endpoint_function(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/fastapi/routing.py", line 163, in run_endpoint_function
return await dependant.call(**values)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/dcef/backend/fastapi/app.py", line 66, in get_guilds
whitelisted_guild_ids = get_whitelisted_guild_ids()
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/dcef/backend/fastapi/helpers.py", line 18, in get_whitelisted_guild_ids
whitelisted_guild_ids = collection_config.find_one({"key": "whitelisted_guild_ids"})["value"]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/pymongo/collection.py", line 1459, in find_one
for result in cursor.limit(-1):
File "/usr/local/lib/python3.11/dist-packages/pymongo/cursor.py", line 1248, in next
if len(self.__data) or self._refresh():
^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/pymongo/cursor.py", line 1139, in _refresh
self.__session = self.__collection.database.client._ensure_session()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/pymongo/mongo_client.py", line 1740, in _ensure_session
return self.__start_session(True, causal_consistency=False)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/pymongo/mongo_client.py", line 1685, in __start_session
self._topology._check_implicit_session_support()
File "/usr/local/lib/python3.11/dist-packages/pymongo/topology.py", line 538, in _check_implicit_session_support
self._check_session_support()
File "/usr/local/lib/python3.11/dist-packages/pymongo/topology.py", line 554, in _check_session_support
self._select_servers_loop(
File "/usr/local/lib/python3.11/dist-packages/pymongo/topology.py", line 238, in _select_servers_loop
raise ServerSelectionTimeoutError(
pymongo.errors.ServerSelectionTimeoutError: 127.0.0.1:27017: [Errno 111] Connection refused, Timeout: 30s, Topology Description: <TopologyDescription id: 653e9b4525d89af64a3da447, topology_type: Unknown, servers: [<ServerDescription ('127.0.0.1', 27017) server_type: Unknown, rtt: None, error=AutoReconnect('127.0.0.1:27017: [Errno 111] Connection refused')>]>
2023/10/29 17:53:32 [error] 19#19: *3 open() "/dcef/cache/preprocess/fonts/ggsans-normal-400.woff2" failed (2: No such file or directory), client: [CENSORED], server: localhost, request: "GET /data/fonts/ggsans-normal-400.woff2 HTTP/1.1", host: "[CENSORED]:21011", referrer: "http://[CENSORED]:21011/_app/immutable/assets/_layout-92ac35d3.css"
2023/10/29 17:53:32 [error] 19#19: *4 open() "/dcef/cache/preprocess/fonts/ggsans-normal-700.woff2" failed (2: No such file or directory), client: [CENSORED], server: localhost, request: "GET /data/fonts/ggsans-normal-700.woff2 HTTP/1.1", host: "[CENSORED]:21011", referrer: "http://[CENSORED]:21011/_app/immutable/assets/_layout-92ac35d3.css"
this may be out of scope, but sometimes there are a bunch of messages in a channel that you don't want to see/preserve(15k posts that are just variants of "hey/morning/goodnight")
some way of using this(instead of manually editing json/deleting images) to curate a channel would be helpful
i guess this would be best done by having it make a copy before you modify it, and/or leaving it up to the user if they want to make a backup or if they want to edit the "master" archive.
goes up and down a little at random even on no input, making it hard to read. happens with assets disabled too
Wanted to let you know that in the coming months, I will be forking this project to use ElasticSearch search, and also search text attachments. If this is something you are interested in integrating into the main branch, please let me know!
The guild is opened and you can browse its messages
The program gets frozen on startup. After a while it unfreezes and you can click any guilds you have, but clicking on the big guild shows "500 Internal Error"
Is there any relevant information in logs? If so, please provide them:
I don't think so. But here https://pastebin.com/xRZWujwJ
If you provide full logs, please make sure to remove any sensitive information from them.
Hi! I love this front-end version, but it would be amazing if it showed the channels and categories in the original order (not alphabetically). Since the Exporter already shows them in the export options as originally sorted in Discord, I guess it's a front-end thing the fact that those are showing alphabetically. I tried renaming the json files by adding numbers to change the sorting view, but it didn't work (it keeps working and showing as previously).
It would be amazing if there was an option to sort the servers as preferred too.
Can this be possible?
I have an raspberry pi 4 (arm64) and the docker container exits because of wrong cpu arch
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
exec /usr/local/bin/docker-entrypoint.sh: exec format error
is there a workaround?
Servers appearing on the side
There are only the HOME and Settings buttons in the app
--version): v2.9.0How did you export the data?
--format): Json--media): Yes--markdown): YesIs there any relevant information in logs? If so, please provide them:
2024/03/08 13:51:27 [error] 44848#43784: *6 connect() failed (10061: No connection could be made because the target machine actively refused it) while connecting to upstream, client: 127.0.0.1, server: localhost, request: "GET /api/guilds HTTP/1.1", upstream: "http://127.0.0.1:58000/guilds", host: "127.0.0.1:21011", referrer: "http://127.0.0.1:21011/"
On channels with fewer messages, the client immediately jumps to the latest message archived. When using the search feature and clicking on a message to jump to, the client loads the message and its context immediately. This should occur regardless of number of messages in the channel.
For channels around 100k-140k messages, the client visibly stutters and stays blank for a second. The logs show a few (~4) http requests to the messages endpoint.
For larger channels, around 150k+, the channel is unusable, as the client starts at the very top of the channel and attempts to continuously scroll down, spamming post requests to the server. Changing the "default scroll position" to "top" prevents this, and allows scrolling normally, but the bottom of the channel is still completely unreachable. Jumping to searched messages results in the same behaviour as clicking on the channel with the "bottom" scroll position - incredibly long waiting times as the client scrolls through messages.
--version): v2.39.1 (I don't have such large backups from v2.40+)How did you export the data?
--format): Json--media): Yes--markdown): No--reuse-mediaIs there any relevant information in logs? If so, please provide them:
Possibly relevant, for a channel of 140k messages, just clicking on the channel for the first time with scroll position set to "bottom" results in multiple (2-5) post message requests, where smaller channels only request once:
INFO: 127.0.0.1:0 - "GET /search-categories HTTP/1.1" 200 OK
get_message_ids() cache hit - channel id 000000xxxxxxxxxxxxxxxxxx
INFO: 127.0.0.1:0 - "GET /message-ids?guild_id=000000xxxxxxxxxxxxxxxxxx&channel_id=000000xxxxxxxxxxxxxxxxxx HTTP/1.1" 200 OK
INFO: 127.0.0.1:0 - "GET /search-categories HTTP/1.1" 200 OK
INFO: 127.0.0.1:0 - "POST /messages HTTP/1.1" 200 OK
INFO: 127.0.0.1:0 - "POST /messages HTTP/1.1" 200 OK
INFO: 127.0.0.1:0 - "POST /messages HTTP/1.1" 200 OK
Jumping to a searched message in the middle of the channel sends a further 5-6 requests.
The logs for a channel with 180k messages are similar, except the message requests keep being spammed ad nauseam. I got a few hundred before ether the client or server seemingly gave up and stopped requesting/responding to requests, and refused to load more messages.
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.