[SOLVED] LDB database size > 4GB

Come here to share your tips and tricks for using Samba4
Locked
gtirmont
Messages: 5
Registration: Sep 21, 2018 - 4:54 p.m.

December 5, 2018 - 5:24 PM

Good morning,

The file size /var/lib/samba/private/sam.ldb.d/DC%3DADS,DC%3DUNIV-LILLE,DC%3DFR.ldb Our domain ads.univ-lille.fr has already exceeded the 4G limit :?

Code: Select all

-rw------- 1 root root 4294967295 déc.   5 17:14 /var/lib/samba/private/sam.ldb.d/DC%3DADS,DC%3DUNIV-LILLE,DC%3DFR.ldb
We have 85,000 users, but we can restrict access by only allowing truly active users on this domain.
I only get this error when I try to delete a user:

Code: Select all

ltdb: tdb(/var/lib/samba/private/sam.ldb.d/DC%3DADS,DC%3DUNIV-LILLE,DC%3DFR.ldb): tdb_expand overflow detected current map_size[4294967295] size[9108]!

../source4/dsdb/samdb/ldb_modules/repl_meta_data.c:4447: Failed to rename object from 'CN=test01,OU=IPTA,OU=Autres,OU=ULilleRessources,DC=ads,DC=univ-lille,DC=fr' to 'CN=test01\0ADEL:9ff44ef0-221e-4287-b591-dc330eb76d80,CN=Deleted Objects,DC=ads,DC=univ-lille,DC=fr' - ldb_wait from ../source4/dsdb/samdb/ldb_modules/util.c:499 with LDB_WAIT_ALL: Operations error (1)
ERROR(ldb): Failed to remove user "test01" - ldb_wait from ../source4/dsdb/samdb/ldb_modules/util.c:499 with LDB_WAIT_ALL: Operations error (1)
  File "/usr/lib/python2.7/dist-packages/samba/netcmd/user.py", line 455, in run
    samdb.delete(user_dn)
We can no longer add objects (user, computer...)

Do you have a solution for reducing the database size? Even if it means deleting objects (users, for example)?

Thank you so much for your help
User avatar
dcardon
WAPT Expert
Messages: 1930
Registration: June 18, 2014 - 09:58
Location: Saint Sébastien sur Loire
Contact :

December 6, 2018 - 12:30

Hello gtirmont,
gtirmont wrote: Dec 5, 2018 - 5:24 PM Good morning,

The file size /var/lib/samba/private/sam.ldb.d/DC%3DADS,DC%3DUNIV-LILLE,DC%3DFR.ldb Our domain ads.univ-lille.fr has already exceeded the 4G limit :?

Code: Select all

-rw------- 1 root root 4294967295 déc.   5 17:14 /var/lib/samba/private/sam.ldb.d/DC%3DADS,DC%3DUNIV-LILLE,DC%3DFR.ldb
We have 85,000 users, but we can restrict access by only allowing truly active users on this domain.
I only get this error when I try to delete a user:

Code: Select all

ltdb: tdb(/var/lib/samba/private/sam.ldb.d/DC%3DADS,DC%3DUNIV-LILLE,DC%3DFR.ldb): tdb_expand overflow detected current map_size[4294967295] size[9108]!

../source4/dsdb/samdb/ldb_modules/repl_meta_data.c:4447: Failed to rename object from 'CN=test01,OU=IPTA,OU=Autres,OU=ULilleRessources,DC=ads,DC=univ-lille,DC=fr' to 'CN=test01\0ADEL:9ff44ef0-221e-4287-b591-dc330eb76d80,CN=Deleted Objects,DC=ads,DC=univ-lille,DC=fr' - ldb_wait from ../source4/dsdb/samdb/ldb_modules/util.c:499 with LDB_WAIT_ALL: Operations error (1)
ERROR(ldb): Failed to remove user "test01" - ldb_wait from ../source4/dsdb/samdb/ldb_modules/util.c:499 with LDB_WAIT_ALL: Operations error (1)
  File "/usr/lib/python2.7/dist-packages/samba/netcmd/user.py", line 455, in run
    samdb.delete(user_dn)
We can no longer add objects (user, computer...)

Do you have a solution for reducing the database size? Even if it means deleting objects (users, for example)?

Thank you so much for your help
We secured funding for the development of the LMDB backend for Samba 4.9, which removes the 4GiB limit that we are familiar with.

You would therefore need to switch to Samba 4.9 LMDB. But to do this, you would first need to integrate a new 4.9 LMDB domain controller, which isn't possible since you can no longer create objects...

For your information, deleting a user doesn't actually delete the entry, but moves it to CN=Deleted Objects. Therefore, no space is freed up until the tombstone expiration has passed and Active Directory has flushed the object. Furthermore, when the object is flushed, TDB also needs to reclaim the space it freed up.

Finally, one thing you can try is to run a `tdbbackup` and restore from that database (this reclaims the freed space, a bit like a `VACUUM` in PostgreSQL). But if the database is full, it might not work. You could try `ldbdel`, but I think that's still a bit too high-level. You'd need to look at the TDB records corresponding to the LDB abstraction layer for objects you can delete.

Given the size of your university, I encourage you to contact the sales team for support! In the very short term, TDB cleanup is probably the best option, but you should upgrade to 4.9 LMDB fairly quickly. We've done quite intensive testing on 4.9 LDMB and it delivers the expected results. However, being rather conservative with something as critical as Active Directory, we're still on 4.8 for the vast majority of our clients.

Sincerely,

Denis
Denis Cardon - Tranquil IT
Share your experiences on WAPT! Send us your blog and article URLs in the "Your Opinion of the forum, and we'll feature them on the WAPT
gtirmont
Messages: 5
Registration: Sep 21, 2018 - 4:54 p.m.

December 6, 2018 - 2:38 PM

Thank you so much for your reply!

I had just validated tdbbackup on our development infrastructure (a single DC) and it worked perfectly.
I was about to do the same thing on our test infrastructure (2 DCs). Is there a specific procedure to avoid breaking replication?

Best regards,

Gaétan
gtirmont
Messages: 5
Registration: Sep 21, 2018 - 4:54 p.m.

December 19, 2018 - 3:40 PM

Hello,

I performed a database backup (tdbbackup) and it went from 4.1GB to 1.5GB. I demoted the second DC beforehand to avoid replication issues.
I then reinstalled the second DC and joined it with the `--domain-critical-only` option to replicate only the minimum initially, and the rest of the objects later. Without this option, replication happens immediately, and with a 1.5GB database, it takes time and results in an LDAP timeout (https://git.samba.org/?p=samba.git;a=co ... 079a292079).

Today, we can create objects again, and replication is working.
Please mark this thread as resolved. ;)

Best regards,
Gaétan
Locked