Can't pickle <type 'thread.lock'> when using python multiprocess.pool.map_async() -

i'm trying use map_async on class method , error: picklingerror: can't pickle <type 'thread.lock'>: attribute lookup thread.lock failed code :

def _pickle_method(method):     func_name = method.im_func.__name__     obj = method.im_self     cls = method.im_class     cls_name = ''     if func_name.startswith('__') , not func_name.endswith('__'):         cls_name = cls.__name__.lstrip('_')     if cls_name:         func_name = '_' + cls_name + func_name     return _unpickle_method, (func_name, obj, cls)  def _unpickle_method(func_name, obj, cls):     cls in cls.mro():         try:             func = cls.__dict__[func_name]         except keyerror:             pass         else:             break     return func.__get__(obj, cls)  copy_reg.pickle(types.methodtype, _pickle_method, _unpickle_method)  class myclass(object):     def submit(self,cmd):         subprocess.call(cmd, shell=true)      def runtest(self):         cmds = []         in range(50):             cmd = createcmd(self)             cmds.append(cmd)         self.pool.map_async(self.submit, cmds)       def main(self):         self.pool = mp.pool         while true:            runtest(self)  if __name__ == "__main__":     myclass() 

when submit outside class works error. also, myclass has more methods , attributes didn't wrote, 1 of them logger, problem?

so built code alternate imports, notably dill instead of pickle. used fork of multiprocessing called pathos.multiprocessing uses dill. can pickle class methods , bound methods. ignored whole part teaching copy_reg how pickle modules, because dill can already.

i had make modifications code, well, because didn't work. had make createcmd function, because didn't give one. also, code as-is, launch multiprocessing jobs… never results because don't ask them. trying do?

anyway, here's code yours, works. still doesn't give results worth anything, except show pickles , code runs. please post code can run, , throw error reporting.

>>> import dill pickle >>> import subprocess  >>> pathos.multiprocessing import processingpool pool >>>  >>> def createcmd(cmd): ...   return 'sleep {0}'.format(cmd) >>> >>> class myclass(object): ...   def submit(self, cmd): ...     subprocess.call(cmd, shell=true) ...   def runtest(self): ...     cmds = [] ...     in range(50): ...       cmd = createcmd(i) ...       cmds.append(cmd) ...     self.pool.amap(self.submit, cmds) # equivalent map_async ...   def main(self):  ...     self.pool = pool() ...     self.runtest() ...  >>> pickle.loads(pickle.dumps(myclass)) <class '__main__.myclass'> >>> pickle.loads(pickle.dumps(myclass.runtest)) <unbound method myclass.runtest> >>> x = myclass() >>> pickle.loads(pickle.dumps(x.runtest)) <bound method myclass.runtest of <__main__.myclass object @ 0x10d015b10>> >>> x.main() >>> x.submit('sleep 1') >>> # use get result… 'sleep' felt script  >>> res = x.pool.amap(x.submit, (createcmd(i) in range(10))) >>> res.get() [none, none, none, none, none, none, none, none, none, none] 

anyway, if want dill or pathos, can them at: https://github.com/uqfoundation

by way, if wanted pickle thread lock, can too.

>>> import dill pickle >>> import threading >>> lock = threading.lock() >>>  >>> pickle.loads(pickle.dumps(lock)) <thread.lock object @ 0x10c534650>