We accidentally deleted the directory rm -rf /data/db
which was our MongoDB path, and thanks to
extundelete, we recovered it and got the directory /data/db.
Here are our files in the directory, and the files was generated under MongoDB version 3.4.
Folder diagnostic.data:
Folder journal:
Step 1: try to run mongod as normal
a) We ran mongod --port 27017 --dbpath /data/db --bind_ip_all and mongo, and expected there should have a user-defined database wecaXX, But, it did not show up.
> show dbs
admin 0.000GB
config 0.000GB
local 0.000GB
In Robo3T
b) Then I tried to run mongod --port 27017 --dbpath /data/db --bind_ip_all --repair. The result was:
2019-03-25T14:10:02.170+0800 I CONTROL [main] Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'
2019-03-25T14:10:02.191+0800 I CONTROL [initandlisten] MongoDB starting : pid=23018 port=27017 dbpath=/data/db 64-bit host=iZj6c0pipuxk17pb7pbaw0Z
2019-03-25T14:10:02.191+0800 I CONTROL [initandlisten] db version v4.0.7
2019-03-25T14:10:02.191+0800 I CONTROL [initandlisten] git version: 1b82c812a9c0bbf6dc79d5400de9ea99e6ffa025
2019-03-25T14:10:02.191+0800 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.2g 1 Mar 2016
2019-03-25T14:10:02.191+0800 I CONTROL [initandlisten] allocator: tcmalloc
2019-03-25T14:10:02.191+0800 I CONTROL [initandlisten] modules: none
2019-03-25T14:10:02.191+0800 I CONTROL [initandlisten] build environment:
2019-03-25T14:10:02.191+0800 I CONTROL [initandlisten] distmod: ubuntu1604
2019-03-25T14:10:02.191+0800 I CONTROL [initandlisten] distarch: x86_64
2019-03-25T14:10:02.191+0800 I CONTROL [initandlisten] target_arch: x86_64
2019-03-25T14:10:02.191+0800 I CONTROL [initandlisten] options: { net: { bindIpAll: true, port: 27017 }, repair: true, storage: { dbPath: "/data/db" } }
2019-03-25T14:10:02.191+0800 W STORAGE [initandlisten] Detected unclean shutdown - /data/db/mongod.lock is not empty.
2019-03-25T14:10:02.194+0800 I STORAGE [initandlisten] Detected data files in /data/db created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2019-03-25T14:10:02.194+0800 W STORAGE [initandlisten] Recovering data from the last clean checkpoint.
2019-03-25T14:10:02.194+0800 I STORAGE [initandlisten]
2019-03-25T14:10:02.194+0800 I STORAGE [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2019-03-25T14:10:02.194+0800 I STORAGE [initandlisten] ** See http://dochub.mongodb.org/core/prodnotes-filesystem
2019-03-25T14:10:02.194+0800 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=256M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),
2019-03-25T14:10:02.818+0800 E STORAGE [initandlisten] WiredTiger error (17) [1553494202:818725][23018:0x7f6119074a40], connection: __posix_open_file, 715: /data/db/WiredTiger.wt: handle-open: open: File exists Raw: [1553494202:818725][23018:0x7f6119074a40], connection: __posix_open_file, 715: /data/db/WiredTiger.wt: handle-open: open: File exists
2019-03-25T14:10:02.818+0800 I STORAGE [initandlisten] WiredTiger message unexpected file WiredTiger.wt found, renamed to WiredTiger.wt.1
2019-03-25T14:10:03.832+0800 I STORAGE [initandlisten] WiredTiger message [1553494203:832267][23018:0x7f6119074a40], txn-recover: Main recovery loop: starting at 4/11366912 to 5/256
2019-03-25T14:10:03.832+0800 I STORAGE [initandlisten] WiredTiger message [1553494203:832674][23018:0x7f6119074a40], txn-recover: Recovering log 4 through 5
2019-03-25T14:10:03.898+0800 I STORAGE [initandlisten] WiredTiger message [1553494203:898252][23018:0x7f6119074a40], txn-recover: Recovering log 5 through 5
2019-03-25T14:10:03.964+0800 I STORAGE [initandlisten] WiredTiger message [1553494203:964880][23018:0x7f6119074a40], txn-recover: Set global recovery timestamp: 0
2019-03-25T14:10:03.998+0800 I RECOVERY [initandlisten] WiredTiger recoveryTimestamp. Ts: Timestamp(0, 0)
2019-03-25T14:10:03.999+0800 E STORAGE [initandlisten] WiredTiger error (17) [1553494203:999855][23018:0x7f6119074a40], WT_SESSION.create: __posix_open_file, 715: /data/db/_mdb_catalog.wt: handle-open: open: File exists Raw: [1553494203:999855][23018:0x7f6119074a40], WT_SESSION.create: __posix_open_file, 715: /data/db/_mdb_catalog.wt: handle-open: open: File exists
2019-03-25T14:10:04.000+0800 I STORAGE [initandlisten] WiredTiger message unexpected file _mdb_catalog.wt found, renamed to _mdb_catalog.wt.1
2019-03-25T14:10:04.015+0800 I CONTROL [initandlisten]
2019-03-25T14:10:04.015+0800 I CONTROL [initandlisten] ** WARNING: Access control is not enabled for the database.
2019-03-25T14:10:04.015+0800 I CONTROL [initandlisten] ** Read and write access to data and configuration is unrestricted.
2019-03-25T14:10:04.015+0800 I CONTROL [initandlisten] ** WARNING: You are running this process as the root user, which is not recommended.
2019-03-25T14:10:04.015+0800 I CONTROL [initandlisten]
2019-03-25T14:10:04.015+0800 I CONTROL [initandlisten]
2019-03-25T14:10:04.015+0800 I CONTROL [initandlisten] ** WARNING: soft rlimits too low. rlimits set to 3824 processes, 65535 files. Number of processes should be at least 32767.5 : 0.5 times number of files.
2019-03-25T14:10:04.020+0800 I STORAGE [initandlisten] createCollection: admin.system.version with provided UUID: 47d8713d-ac61-4081-83bf-60209ad60a7d
2019-03-25T14:10:04.034+0800 W ASIO [initandlisten] No TransportLayer configured during NetworkInterface startup
2019-03-25T14:10:04.036+0800 I COMMAND [initandlisten] setting featureCompatibilityVersion to 4.0
2019-03-25T14:10:04.036+0800 I STORAGE [initandlisten] repairDatabase admin
2019-03-25T14:10:04.037+0800 I STORAGE [initandlisten] Repairing collection admin.system.version
2019-03-25T14:10:04.040+0800 I STORAGE [initandlisten] Verify succeeded on uri table:collection-0-4352287918877967674. Not salvaging.
2019-03-25T14:10:04.048+0800 I INDEX [initandlisten] build index on: admin.system.version properties: { v: 2, key: { _id: 1 }, name: "_id_", ns: "admin.system.version" }
2019-03-25T14:10:04.048+0800 I INDEX [initandlisten] building index using bulk method; build may temporarily use up to 500 megabytes of RAM
2019-03-25T14:10:04.055+0800 I STORAGE [initandlisten] finished checking dbs
2019-03-25T14:10:04.055+0800 I STORAGE [initandlisten] WiredTigerKVEngine shutting down
2019-03-25T14:10:04.056+0800 I STORAGE [initandlisten] Shutting down session sweeper thread
2019-03-25T14:10:04.057+0800 I STORAGE [initandlisten] Finished shutting down session sweeper thread
2019-03-25T14:10:04.140+0800 I STORAGE [initandlisten] shutdown: removing fs lock...
2019-03-25T14:10:04.140+0800 I CONTROL [initandlisten] now exiting
2019-03-25T14:10:04.140+0800 I CONTROL [initandlisten] shutting down with code:0
c) After repairing I re-ran mongod --port 27017 --dbpath /data/db --bind_ip_all, it still showed nothing (the same result as step a)).
Step 2: Use WiredTiger tool
Since that didn't work as I expected, I started to find any other tools or approach that may help me. Here is the link Recovering a WiredTiger collection from a corrupt MongoDB installation which introduces how to use WiredTiger to recover collections. And I decided to have a try.
I created a folder db_backup and put all my files into it. And I created a folder wiredtiger-3.0.0 under db_backup.I installed wiredtiger in folder wiredtiger-3.0.0:
a) To salvage the collection
root@iZj6c0pipuxk17pb7pbaw0Z:/data/db_backup/# cd ./wiredtiger-3.0.0
root@iZj6c0pipuxk17pb7pbaw0Z:/data/db_backup/wiredtiger-3.0.0# ./wt -v -h ../db_for_wt -C "extensions=[./ext/compressors/snappy/.libs/libwiredtiger_snappy.so]" -R salvage collection-23--360946650994865928.wt
WT_SESSION.salvage 100
Based on the article above, it means that we have successfully salvaged this collection.
Error Occured:
b) To dump the collection
root@iZj6c0pipuxk17pb7pbaw0Z:/data/db_backup/wiredtiger-3.0.0# ./wt -v -h ../../db_backup -C "extensions=[./ext/compressors/snappy/.libs/libwiredtiger_snappy.so]" -R dump -f ../collection.dump collection-23--360946650994865928
lt-wt: cursor open(table:collection-23--360946650994865928) failed: No such file or directory
Note that the command above was indeed ended without .wt.
I have checked my directory argument and I found no fault. In the picture, the file collection-23--360946650994865928.wt is right Here.
The snapshot of collection-23--360946650994865928.wt just has been salvaged.
We could see some English or Chinese characters in it. And that data truly is from one of collection of database wecaXX.
Questions:
1) Now, I'm stuck by dumping the collection. Does anyone know what's wrong with that?
2) collection-23--360946650994865928.wt contains our most important data. If we could not restore the whole database, extracting data from that file will be still very useful. However, if we do copy paste manually, not all the characters are well written; they still need to be decrypted. Does anyone know how to do that?
3) is it possible that we did not recover all the files (or the entire body of all the files) under /data/db?
Any comment or answer will be appreciated!!
