Opening a SharePoint 2010 dialog box for list form

If you want to open a window in a dialog from clicking a link then simply use the following JavaScript as the href property.

javascript:var options = {url: '/Lists/MyList/NewForm.aspx?IsDlg=1&ContentTypeId=0x001989BCF16D8914CAF8862B0C6C89E091E', title: 'Showing List'}; void(SP.UI.ModalDialog.showModalDialog(options))

As you can see this is simply taking the script you would usually call and adapting it to work in the href of a hyperlink anchor. It needs to be all on one line and return void when called.

So why might you want to do this? Many reasons. Perhaps you have a list with multiple content types and you wish to open the new item form in a dialog AND change the title from the default “[List Title] – New Item” depending on which content type the item is being created from. The snippet above illustrates this. Or perhaps you wish to open a document library in a dialog. Or even an external site.

Also you can use following script in your code :

public string GetDialogURL(string url, bool refreshOnClose , double width, double height )
        {
            string strDialogURL = "javascript:function demoCallback(dialogResult, returnValue){ SP.UI.Notify.addNotification('Operation Successful!'); SP.UI.ModalDialog.RefreshPage(SP.UI.DialogResult.OK);  }function(){setTimeout('editEffect()', 200)}editEffect(); function editEffect(){" +
                                  " var options = { url:'" + url + "'," +
                                  "tite: 'Move Documents',allowMaximize: true ,showClose: true,width:" + width + ",height:" + height;


            if (refreshOnClose)
            {
                strDialogURL += ",dialogReturnValueCallback: demoCallback";
            }
            strDialogURL += "};SP.UI.ModalDialog.showModalDialog(options);}";
            return strDialogURL;
        }

Check this also :

SharePoint: Opening a 2010 Dialog Box from Quick Launch

Microsoft