xfs
[Top] [All Lists]

Re: [Myricom help #36078] application sleeps indefinitely trying

To: arajekar@xxxxxxxxx
Subject: Re: [Myricom help #36078] application sleeps indefinitely trying to read gm_dma_malloc'ed buffer through O_DIRECT]]
From: Myricom Technical Support <help@xxxxxxxx>
Date: Tue, 30 Aug 2005 18:53:40 -0400
Cc: linux-xfs@xxxxxxxxxxx, ocrete@xxxxxxxxx, Myricom Technical Support <help@xxxxxxxx>
Organization: Myricom, Inc.
Sender: linux-xfs-bounce@xxxxxxxxxxx
User-agent: Mozilla Thunderbird 1.0.2 (X11/20050317)
Ashutosh,

From one of our GM software developers:

I suggest they try the attached patch which should solve their problem.

Basically the incompatibility is that gm-registration of any
"process-private" memory requires the gm driver to lock the
corresponding pages for the duration of the registration (which is the
life of the gm-port for gm_dma_malloc() buffers). During that time
those pages cannot be used for reading through O_DIRECT (because
O_DIRECT also requires obtaining a lock on the pages at some point).

The patch changes gm_dma_malloc() to allocate memory with
map(MAP_SHARED|MAP_ANONYMOUS). The gm-registration of such pages does
not require a lock and should be compatible with any O_DIRECT activity
(or any other activity that we can think of at this point).

Note that this patch potentially introduces a small semantic change
for programs that uses fork(), but should not be a problem as we don't
expect programs using forks() to rely on any specific sharing
semantics for the gm_dma_malloc'ed() buffers.

Please apply this patch to GM-2 and let us know how this works for you.

Thanks,
Susan


--
Susan Blackford
Member of Technical Staff
Myricom Inc.
Index: libgm/gm_dma_malloc.c
===================================================================
RCS file: /repository/gm/libgm/gm_dma_malloc.c,v
retrieving revision 1.8.22.1
diff -u -r1.8.22.1 gm_dma_malloc.c
--- libgm/gm_dma_malloc.c       28 May 2004 19:09:10 -0000      1.8.22.1
+++ libgm/gm_dma_malloc.c       30 Aug 2005 20:55:55 -0000
@@ -246,6 +246,26 @@
 }
 #endif /* GM_KERNEL */
 
+
+#if GM_KERNEL || !GM_OS_LINUX
+#define gm_alloc_pages_for_dma gm_alloc_pages
+#define gm_alloc_free_pages_for_dma gm_free_pages
+#else
+#include <sys/mman.h>
+
+void *gm_alloc_pages_for_dma(gm_size_t length)
+{
+  void * ptr = mmap(0, length, PROT_READ | PROT_WRITE, MAP_SHARED | 
MAP_ANONYMOUS, -1, 0);
+  return (ptr == MAP_FAILED) ? 0 : ptr;
+}
+
+void gm_free_pages_for_dma(void *addr, gm_size_t length)
+{
+  munmap(addr, length);
+}
+#endif
+
+
 /****************************************************************
  * Environment abstraction
  *
@@ -272,7 +292,7 @@
   
     /* allocate some pages */
     
-    addr = gm_alloc_pages (length);
+    addr = gm_alloc_pages_for_dma (length);
     if (addr == 0)
       {
        status = GM_OUT_OF_MEMORY;
@@ -300,7 +320,7 @@
     GM_RETURN_STATUS (GM_SUCCESS);
     
   abort_with_pages:
-    gm_free_pages (addr, length);
+    gm_free_pages_for_dma (addr, length);
   abort_with_nothing:
     GM_RETURN_STATUS (status);
   }
@@ -447,7 +467,7 @@
 #if !GM_KERNEL && GM_CAN_REGISTER_MEMORY
   {
     gm_deregister_memory (p, ptr, length);
-    gm_free_pages (ptr, length);
+    gm_free_pages_for_dma (ptr, length);
   }
 #elif !GM_KERNEL && !GM_CAN_REGISTER_MEMORY
   {

<Prev in Thread] Current Thread [Next in Thread>
  • Re: [Myricom help #36078] application sleeps indefinitely trying to read gm_dma_malloc'ed buffer through O_DIRECT]], Myricom Technical Support <=