Documentum Bulk Export Tool

For some time now I’ve had this idea rattling around in my head that there should be a simple way to export content and metadata from the Content Server in a manner that allows for quick access to both the exported content and the metadata. Drag & Drop or Export from Webtop only gets you the content files. Dump & Load is a major pain, inconsistent, and too opaque. DAR files can also be problematic and too opaque.  I wanted a way to quickly export the content in its native format and the relevant metadata as XML. These pairs of files would then contain everything I needed to know about the content. I could visually verify them on the file system; I could open the content with a double-click; I could easily examine the metadata — or change it; I could easily archive it to CD or a ZIP file; or I could import it into another CMS.  (I think Records Manager does something like this when you disposition files to NARA as XML.)  Anyway, I finally converted the idea to code. Here is the Java/DFC export tool if you are interested.

To run it, configure the export.properties file to contain the necessary login credentials for your repository, and a query to gather the files you want to export. In addition, specify a location for the exported content on your file system. Here is my sample export.properties file.

export.query=select * from dm_sysobject where folder('/Temp',descend)
export.user=dmadmin
export.password=dmadmin
export.repo=repo1
export.path=c:/Temp/Export
export.log=c:/Temp/Export/export.log

Notes:

  • After the export tool runs once it will encrypt the password so it doesn’t sit around unprotected.
  • The query must return r_object_id.

Once configured, run the export.bat batch file. The batch file does its best to create the correct class path, but you might have to help it if your Documentum environment is not installed in the default manner.

The export tool will export all of the objects it gathers from running the configured query. Each content object is exported to a file name that is its r_object_id (an easy way to avoid any file name collisions), followed by the DOS extension for its content type. The accompanying metadata is similarly named, followed with .metadata.xml. All files are exported to the path you specify in the export.properties file.   Note that no folder structures are created to mirror the structure in the repository. That is, the export directory will contain a “flat” file structure (that’s why avoiding file name collisions was important). Right now, the export tool can handle dm_sysobjects, dm_folders, virtual documents, and content-less objects. Since the export tool will export whatever the query returns, you can export versions by specifying ‘(all)‘ in the query, and subfolder structures by using the FOLDER(*,'descend') function.

Here is an example .metadata.xml file.

export_xml

Notes:

  • This particular file is a virtual document, and it’s child nodes are listed in the <vd_children> element.
  • By default, the export tool exports metadata values for:  object_name, object_type, title, subject, acl_domain, acl_name, owner, version and a_content_type) (if applicable).
  • If the type is a custom type (like this example is), values for all custom metadata properties will also be exported (note sr_cat_no, sr_lot_no, and sr_expire above).

If the export tool encounters an object that appears to be a custom type, it will export a .typedef.xml file that contains definitions for all its custom attributes.

Here is an example of a .typedef.xml file.

type_xml

It’s not perfect, but it works for my purposes.  Perhaps one day I will write an import tool that will read the XML and content files and import them into another repository (e.g., think simple migration tool). However, I suspect that the import process will be a lot more complicated than the export process. Keep an eye here, it might show up one day.

Advertisements

About Scott
I have been implementing Documentum solutions since 1997. In 2005, I published a book about developing Documentum solutions for the Documentum Desktop Client (ISBN 0595339689). In 2010, I began this blog as a record of interesting and (hopefully) helpful bits of information related to Documentum, and as a creative outlet.

54 Responses to Documentum Bulk Export Tool

  1. Congratulations you’ve reinvented (at least in part) OpenMigrate. I guess there is still some value in writing your own code though. At least you will know how it works and can have documentation for it, unlike OpenMigrate.

    Like

  2. Dirk says:

    interesting tool…. mail me I would like to have a conversation about this

    Like

  3. David Lloyd says:

    It’d be especially useful to have a properties file that supported multiple query/destination pairings, so that it could execute a “batch” of exports. At that point, the properties file would probably make more sense as XML itself, but it would have a LOT of additional utility!

    Like

  4. Pingback: New Documentum Enterprise Migration Appliance (EMA) Info Available | dm_misc: Miscellaneous Documentum Information

  5. Nirav Modi says:

    This is the great tool. But I want to download physical document. Can you please help me for download physical document from documentum.

    Like

    • Scott says:

      I’m not sure I completely understand your question. The API to download/export a document from a Documentum repository is in the IDfSysObject class. The method is getFile(String path).

      Like

  6. Malcolm says:

    Hi Mark,

    Great tool – thanks! However, I had to change a few things to get it to work with our Documentum 5 server, and though I do not anticipate any problems with the changes I made, I would like to know if there are possibly any other issues I may have with such an old version? I.e. will it still function given the changes made below?

    1. I decompiled the classes in the JAR file, and discarded the decompiled version of JDOM, as it was impossible to recompile (I think Unicode character constants were incorrectly decompiled, rendering the source unusable) and replaced it with a precompiled jdom-2.0.5.jar, alterng the CLASSPATH in export.bat to include it.
    2. Documentum 5 DFC doesn’t have the RegistryPasswordUtils, so I comemnted out

    import com.documentum.fc.tools.RegistryPasswordUtils;

    in Utils.Java
    3. For the same reason, in checkPassword, commented out the if (!str2.startsWith(“DM_ENCR_TEXT=”)) { … block

    4. Commented out the WritePasswordToPropertyFile function, and removed the single call to it.

    5. In dmRecordSet.java, the calls to DFException don’t work in version 5. They expect an integer and string as minimal parameters (win’t work with just a string). Since, when you hit an exception the program will halt, I just filled in the number ‘2’ for the integer parameter.

    I have encountered no problems so far with the tool after these changes. Is there anything I might have missed?

    Like

    • Scott says:

      Malcolm, thanks for the feedback on the QuikDIE export tool. I guess I should have made a note somewhere that I only tested the tool on DCTM 6.5, 6.7 and D7. That said, I don’t see the changes that you made as critical. It seems that most of them revolve around encrypting the password in the export.properties file if it isn’t already. Not a big deal if you can live with it. The change to the DmRecordSet seems minimal as well. Again, not tested on D5. I don’t anticipate you having any problems running the tool on D5, it sticks to basic DFC calls, queries, and attrs. Run it on a small set just to be sure!

      Like

      • Malcolm says:

        Thanks – I thought I should check with the author though!

        We have tested small datasets and seen no discrepancies and – as you say – the changes made shouldn’t have an effect on the core functionality. The unencrypted passwords, I can live with, since (you might have guessed from the fact it’s version 5…) this is for an export from a legacy system.

        Like

  7. Malcolm says:

    Hi Scott,

    There is one major omission in the QuikDIE tool – it does not export creation/modification dates for files. I am having a look at the source and the DFC references to try and figure out how that could be added …

    Like

    • Malcolm says:

      Figured that out, longer post with changes to follow later. [Extending the import side to take in that metadata will be someone else’s job … I don’t need to import, and I am afraid I don’t have the time to add it in]

      Makes the tool much more useful now.

      Like

      • Malcolm says:

        In ExportObj.java, function buildAttrExportString(), I added the following in between the StringBuilder calls for ‘subject’ and ‘acl’:

        localStringBuilder.append(String.format(str, new Object[] { “creator_name”, “string”, this.m_sObj.getCreatorName() }));
        localStringBuilder.append(String.format(str, new Object[] { “creation_date”, “string”, this.m_sObj.getCreationDate().asString(“yyyyMMddHHmmss”) }));
        localStringBuilder.append(String.format(str, new Object[] { “last_modified_by”, “string”, this.m_sObj.getModifier() }));
        localStringBuilder.append(String.format(str, new Object[] { “modification_date”, “string”, this.m_sObj.getModifyDate().asString(“yyyyMMddHHmmss”) }));

        The string constant used was created to export it in ISO format (to avoid US/Europe confusion over where the month is…).

        Import is a different matter. I haven’t looked at that side of things.

        Liked by 1 person

  8. Scott says:

    You got it right. I’ll merge these changes into the code base for a future release. I may also look at parsing the select statement to determine what attrs to export. In that way, you could just do “select create_name, modification_date from dm_document where…”.

    Like

  9. Malcolm says:

    Hi Mark,

    We have now begun successfully using the tool to export items from Documentum!

    However, if you ever do consider a rewrite, there’s a flaw in the export functionality; it doesn’t produce properly escaped XML – meaning that XML parsers choke as soon as they find an & sign in an object name. I worked around this the lazy way (took someone else’s code for creating properly formatted XML and put it into the source); I guess the proper way would be to get it to use the XML export routines, but that would have basically become a rewrite using tools I’m not familiar with (and possibly getting it wrong). But, it works brilliantly!

    Thanks very much. You saved us a whole lot of bother.

    Like

    • Scott says:

      Thanks for the valuable feedback. You’re right, my test cases did not include any unescaped XML characters. I will certainly fix that as well as add the attributes you suggested above in the next release. Now I just need the time to work on it…. Thanks again.

      Like

  10. Pingback: Documentum Export Tool v1.3 | dm_misc: Miscellaneous Documentum Information

  11. Rombaut says:

    Hi,

    I tried the download page for the QuikDIE-export-v1.3.zip file, but no succes. Do I need prior to register to box.com ?
    How can I download this?
    Thx.

    Like

  12. Jeremy says:

    Hi Scott, This tool seemed to be exactly what I am looking for, but when I attempt to run it I get the following.

    I have tried a number of things with no luck, wondering if you might have suggestions.

    Thank you for any comments.

    Exception in thread “main” java.lang.UnsupportedClassVersionError: Bad version number in .class file
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:620)
    at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:124)
    at java.net.URLClassLoader.defineClass(URLClassLoader.java:260)
    at java.net.URLClassLoader.access$100(URLClassLoader.java:56)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:195)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:268)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
    at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)

    Like

    • Scott says:

      Hmmm. What version of Java are you running? I’m pretty sure this was compiled with 1.6, but I don’t recall which patch version. Also, the latest build is here: https://msroth.wordpress.com/2014/04/27/documentum-export-tool-v1-3/

      Like

      • Jeremy says:

        It was trying to run in 1.5 – I pointed it to 1.6 and got further, but now I am seeing:

        Exception in thread “main” java.lang.NoClassDefFoundError: org/apache/commons/lang/StringEscapeUtils
        at com.dm_misc.QuikDIE.Utils.cleanXML(Utils.java:444)
        at com.dm_misc.QuikDIE.ExportObj.buildAttrExportString(ExportObj.java:206)
        at com.dm_misc.QuikDIE.ExportObj.exportDocumentMetadata(ExportObj.java:167)
        at com.dm_misc.QuikDIE.ExportObj.exportContent(ExportObj.java:133)
        at com.dm_misc.QuikDIE.Export.run(Export.java:221)
        at com.dm_misc.QuikDIE.Export.main(Export.java:128)
        Caused by: java.lang.ClassNotFoundException: org.apache.commons.lang.StringEscapeUtils
        at java.net.URLClassLoader$1.run(Unknown Source)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(Unknown Source)
        at java.lang.ClassLoader.loadClass(Unknown Source)
        at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
        at java.lang.ClassLoader.loadClass(Unknown Source)
        … 6 more

        trying to figure out which jar file that class would be… 🙂

        Like

      • Scott says:

        Make sure the entire DFC class path is included. There is a commons-lang-*.jar file that that the DFC is looking for.

        Like

  13. Jeremy says:

    This is what I have my CLASSPATH set to:

    SET CLASSPATH=C:\Progra~1\Documentum\dctm.jar;D:\Documentum\config;C:\Documentum\product\6.5\dctm-server.jar;C:\Progra~2\Java\jre6\lib\tools.jar;C:\Progra~2\Java\jre6\lib\rt.jar;C:\Documentum\dba\java_methods;Y:\test\commons-codec-1.10-bin\commons-codec-1.10\commons-codec-1.10.jar

    It downloads the first file and throws the exception when it is creating the metadata file

    metadata file ==> 090234138001fe4f.metadata.xml
    Exception in thread “main” java.lang.NoClassDefFoundError: org/apache/commons/lang/StringEscapeUtils
    at com.dm_misc.QuikDIE.Utils.cleanXML(Utils.java:444)
    at com.dm_misc.QuikDIE.ExportObj.buildAttrExportString(ExportObj.java:206)
    at com.dm_misc.QuikDIE.ExportObj.exportDocumentMetadata(ExportObj.java:167)
    at com.dm_misc.QuikDIE.ExportObj.exportContent(ExportObj.java:133)
    at com.dm_misc.QuikDIE.Export.run(Export.java:221)
    at com.dm_misc.QuikDIE.Export.main(Export.java:128)
    Caused by: java.lang.ClassNotFoundException: org.apache.commons.lang.StringEscapeUtils
    at java.net.URLClassLoader$1.run(Unknown Source)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(Unknown Source)
    at java.lang.ClassLoader.loadClass(Unknown Source)
    at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
    at java.lang.ClassLoader.loadClass(Unknown Source)

    Thank you for any insight!
    Jeremy

    Like

    • Scott says:

      Yep, I don’t see the commons-lang-2.4.jar file in the class path. It is usually in the same directory as the dfc.jar (in your case probably c:\progra-1\Documentum\shared). Try adding the commons-lang-2.4.jar or the whole directory to the class path and see if that clears it up. QuikDie is trying to clean any unallowable characters (e.g. \ ) out of the object name before it writes the XML file.

      Like

      • Tibi says:

        Hi Scott,

        I’m stuck at the very same point as Jeremy. I have JRE 7.2 installed. Adding the commons-lang.jar to the lib-directories and to the paths do not make any difference. My paths look like this:

        CLASSPATH: C:\Program Files\Documentum\dctm.jar;C:\Documentum\config;C:\Program Files\Documentum\Shared\commons-lang-2.4.jar;C:\Program Files (x86)\Java\jre7\bin;C:\Program Files (x86)\Java\jre7\lib;C:\Program Files\Documentum\Shared;

        PATH: C:\Program Files\Documentum\Shared;D:\oracle\product\11.2.0\client_1;D:\oracle\product\10.2.0\client_1;%SystemRoot%\system32;%SystemRoot%;%SystemRoot%\System32\Wbem;%SYSTEMROOT%\System32\WindowsPowerShell\v1.0\;C:\Program Files (x86)\Novell\ZENworks\bin;c:\Program Files\Novell\SecureLogin\;C:\Program Files\WIDCOMM\Bluetooth Software\;C:\Program Files\WIDCOMM\Bluetooth Software\syswow64;C:\Program Files\Microsoft SQL Server\110\Tools\Binn\;C:\Program Files (x86)\Java\jre7\bin;C:\Program Files (x86)\Java\jre7\lib;

        Any new ideas?

        Thank you,

        Tibi

        Like

      • Scott says:

        Hmmm. I haven’t run this code with Java 7 so I will need to reconfigure my development environment and give it a try. What version of the DFC are you using? What version of the export tool are you using?

        Like

      • Tibi says:

        Thanks for the hint, using a higher version of Java should imply a higher version of commons-lang. I downloaded commons-lang-2.5.jar en pointed my CLASSPATH to it. That did the trick!

        I am using DFC 6.5 and QuikDIE-export-v1.3.

        My new challenge is to get i_folder_id into the metadata.xml. This query in export.properties does not do what I expected:

        export.query=select *,i_folder_id from pu_document where folder(‘/P2204/Test’,descend)

        Liked by 1 person

      • Scott says:

        Glad you got it working. I posted a new version (1.4) of QuikDIE this morning compiled and tested against Java 7 with a few small improvements. The query you used and your expected results are on the road map for QuikDIE. My intent is to have a set of default attributes that are always exported (likely the ones it exports now) in addition to any other attributes listed in the select statement of the query. I also hope to have the Import module ready in the near future.

        Like

      • Tibi says:

        Hi Scott,

        Version 1.4 works like a charm, thank you a lot! And a_antecedent_id is really useful for the versioning reconstruction. We are busy doing a massive export-job and your tool is just what we needed. Thanks again!

        Tibi

        Like

      • Scott says:

        Awesome! And thanks for the feedback.

        Like

  14. jermey moore says:

    So I assume this can only be used on a windows based content server? I’m running on Linux, and i noticed the config files doesn’t have any option to point to the server.

    THANKS!

    Like

    • Scott says:

      That’s an interesting point. It is true that I developed this against a Windows/SQL/D7 server, but it never occured to me that it was limited to that. Let me see what I can figure out.

      Like

    • Scott says:

      Actually, now that I think about it… pointing to a Content Server is taken care of by the dfc.properties file. The export tool will connect (via the DFC) to whichever Docbroker is configured in the dfc.properties file and login to the Docbase identified in the export.properties file (export.repo variable). As for the export.path and export.log configurations, there is no reason they can’t be Linux style paths (e.g., /home/dmadmin/export). Does that help?

      Like

  15. Pingback: Documentum Bulk Export Tool v1.4 | dm_misc: Miscellaneous Documentum Information

  16. Michelle says:

    Hello. I am not a programmer. I am a researcher who has been given access to a client’s web-base Documentum. Can you please guide me on how to use your export tool? I need to export PDFs from Documentum so I can review them. Many thanks in advance.

    Like

    • Scott says:

      If you are accessing a Docbase using Webtop you can export content directly. If you need/want to bulk export content and metadata you can use QuikDIE. The README-TOO.txt explains how to run the tool. Essentially configure the export.properties file and run the export.bat batch file.

      Like

  17. Jiggs says:

    Hello – I am just a starter and was thinking something similar. Is it possible to share the source code to have better understanding? Thanks

    Like

  18. Pingback: Custom Export Vdoc Widget – Part 1 – paulsundquist

  19. sonali says:

    Hi Scott, We are also looking for the tool to export and import. We are working on an export-import utility using DFC services to allow export of documents (and hierarchy) from PROD and importing to NON PROD but it seems use of tool would be more feasible. Documentum version is 6.7.

    Like

    • Scott says:

      Note that I never completed the import utility. The source code for what I did complete is included in the download file if you want to expand it to meet your purposes.

      Like

  20. Danny says:

    Hi Scott, how would I make this tool work with Desktop 5.3? I get the following error when executing the batch file: Exception in thread “main” java.lang.UnsupportedClassVersionError: com/dm_misc/QuikDIE/Export (Unsupported major.minor version 51.0).
    I’m totally new to the Documentum world and not sure where to start. Any help will be appreciated.

    Like

    • Scott says:

      Hi Danny, the error you are receiving is a Java error. This code was compiled with an older version of Java than you are trying to run it with. You will either need to run it with an older version of Java (sorry I don’t remember what version it was compiled with, but you can look at the dates of the files and compare them with Java release dates), or recompile it.

      Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: